They probably never will, and that is not a criticism of the product. SageMaker Unified Studio is a front door. It is not a runtime, it is not a billing boundary, and it does not replace your IAM design. Teams who adopt it expecting consolidation do get consolidation, but only of attention. Everything underneath stays where it was, on the same services, on the same bill, behind the same permission model.

This post covers what it genuinely replaces, what it leaves untouched no matter how the migration deck was worded, and what it quietly adds to your operational surface that you will own indefinitely. If you are signing off on an adoption plan, the third part is the one that matters.

SageMaker Unified Studio is a front door, not a runtime

Start here, because every surprise downstream follows from this. When you run a query in the SQL editor, Athena or Redshift runs it. When you build a visual ETL job, Glue runs it. When you train a model, SageMaker AI runs it. When you orchestrate something, Airflow runs it. The studio decides who may open which tool, inside which project, against which data. It executes nothing itself.

AWS is explicit about this in its access control documentation: opening a project is equivalent to signing into an account in a specific Region and assuming one of that project’s IAM roles. The tools you then see are the tools that role can reach. There is no new execution layer. There is a new authorisation layer, and a new set of nouns to describe it. Hold that model and the rest is predictable.

What it actually replaces

The pile of standalone studios and query editors

This is the real win, worth stating plainly before the caveats start. AWS assembled the studio by pulling together the separate studios, query editors and visual tools that previously lived inside Athena, EMR, Glue, Redshift, Managed Workflows for Apache Airflow and the existing SageMaker Studio, then added a Bedrock-backed environment for generative AI work.

The problem it solves is not tab count. It is context loss. A data engineer moving from a Glue job, to the table it wrote, to the notebook consuming that table crossed three consoles, three permission surfaces and three mental models of what a “workspace” meant. That handoff is where mistakes lived. Collapsing it into one project boundary is the reason to adopt at all.

Per-team domain sprawl

The older pattern for isolating business units was a SageMaker domain each, then reconciling identity, networking and tagging across all of them by hand. Projects replace most of that. A project is a multi-user boundary with its own roles, storage and members, and it can span accounts, which the old shared-space model could not do cleanly.

The access request spreadsheet

There are two supported ways to give a project access to data: attach IAM permissions directly to the project role, or publish the asset to the catalog and let members subscribe through a request and approval flow.

The second one is the point of the product. If you are only ever going to use the first, you have bought a portal and skipped the governance engine you are now paying to run.

What it doesn’t replace

Your bill, or your ability to explain it

There is no standalone charge for the studio itself. AWS bills you for each underlying service you reach through it, plus usage of SageMaker Catalog, which is metered separately on metadata ingestion and related operations. Nothing consolidates.

What changes, and this is the part that bites, is attribution. Before, an engineer created a Glue job and it inherited whatever tags your pipeline applied. Now a project profile deploys a bundle of resources on that engineer’s behalf, and unless you have deliberately configured project-level custom tags, those resources land in your cost report under a project name that means nothing to finance.

Sort this out before migration. Decide your tag keys, wire them into the project configuration, and confirm they appear in Cost Explorer and the Cost and Usage Report before you onboard a second team. If you run a cost platform such as CloudZero or Vantage on top of the CUR, check that your allocation rules cope with resources whose creator is a service role rather than a person, because that is the shape everything arrives in now.

Your IAM and Lake Formation design

Project roles are ordinary IAM roles. Lake Formation grants are ordinary Lake Formation grants. Bucket policies still apply. If your data access is broken at any of those layers, the studio will surface the same denial you always got, with slightly less context about which principal was being used at the time.

Nobody in the adoption meeting will say “this fixes our permissions mess.” Plenty of people will quietly assume it. Say it out loud early.

Your infrastructure as code

This one catches DevOps teams hardest. AWS ships an open-source CI/CD CLI for the studio that promotes applications across dev, test and production by mapping each pipeline stage to an isolated project and substituting stage-specific values at deploy time. It is a good tool and it slots into GitHub Actions, GitLab CI or Jenkins without much argument.

It also does not create domains or projects. The documentation says so directly and points you at CloudFormation, CDK or Terraform for the infrastructure underneath, and the same limitations page notes that native notebook deployment is not covered. The substrate is still yours to build and version. You get an application deployment tool, not a platform provisioning tool. Plan for both.

SageMaker AI

The original SageMaker was renamed SageMaker AI and remains available on its own. If your team’s work is model training and inference and nothing else, you are allowed to keep using it directly and skip the governance layer entirely. That is a supported choice, not a holdout position.

What SageMaker Unified Studio quietly adds

Here is the section that rarely makes it into the migration plan.

You now operate DataZone, intentionally or not

SageMaker Catalog is built on Amazon DataZone, and the governance model comes with it: domains, domain units, projects, project profiles, blueprints, environments, glossaries, metadata forms, subscription requests. Domain identifiers still carry a dzd prefix, and the console for creating one still lives under the DataZone console path.

You can see the shape of what you own from the CLI, which is worth doing on day one. These commands read the DataZone control plane sitting behind the studio:

# Every domain in the account and Region, with its status
aws datazone list-domains 
  --query 'items[].{id:id,name:name,status:status}' 
  --output table

# Projects inside one domain (domain IDs look like dzd_xxxxxxxx)
aws datazone list-projects 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The project profiles an admin has published, which is the
# real menu of what users can create
aws datazone list-project-profiles 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The environments actually deployed under one project
aws datazone list-environments 
  --domain-identifier dzd_EXAMPLE 
  --project-identifier PROJECT_ID

Run these monthly against a domain someone else administers and you will learn more about your real governance posture than any dashboard tells you.

Project creation is a CloudFormation deploy wearing a friendly hat

Blueprints are backed by CloudFormation templates. Project profiles bundle blueprint configurations, and choosing a profile triggers a stack deployment that provisions the environment behind the project.

This is the invisible failure mode. To a user, project creation is a form and a spinner. To you, it is a stack that can roll back on a missing permission, an unavailable subnet, a service quota or a name collision. When somebody reports that project creation “just doesn’t work,” do not reproduce it in the UI. Read the stack events:

# Find recently failed or rolled-back stacks
aws cloudformation list-stacks 
  --stack-status-filter CREATE_FAILED ROLLBACK_COMPLETE ROLLBACK_IN_PROGRESS 
  --query 'StackSummaries[].{name:StackName,status:StackStatus}' 
  --output table

# Then read the actual reason, which is almost always
# in the first CREATE_FAILED event, not the last
aws cloudformation describe-stack-events 
  --stack-name STACK_NAME 
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' 
  --output table

The first failed resource causes the rollback. Everything after it is noise. This one habit turns most “the platform is broken” tickets into five-minute answers.

A domain-type fork you will live with for a long time

Domains can authenticate through IAM or through IAM Identity Center. AWS positions the Identity Center configuration as the governance-heavy option, preserving individual user identity through the session and supporting the formal publish and subscribe approval workflow with identity-based audit trails. The IAM configuration is positioned toward lighter-weight developer productivity.

Parity between the two has moved over time and is still moving. Before committing an organisation to one, check the current documentation for the specific capabilities you care about, in the Region you are deploying to. Do not take a blog post’s word for it, including this one. This is the decision that is genuinely expensive to reverse, because it is entangled with how every user signs in.

Where adoption actually goes wrong

“It works in my console but not in the project”

This is the single most common support ticket, and it is almost never a bug. The engineer tested with their own federated role, which has broad permissions, then opened a project running under a narrow project role. Two different principals, two different answers.

Confirm which principal you are before debugging anything else. Run this from a terminal inside the project environment, not from your laptop:

aws sts get-caller-identity

If the ARN is not the one you assumed, stop. You have found the problem.

The bill went up after a migration that was supposed to save money

Usually because the migration created things rather than moving them. New project environments provision new resources, catalog metadata ingestion is metered, and per-project orchestration environments are not free just because nobody chose them explicitly. Compare like for like by tag rather than by service total, and give it a full billing cycle before drawing conclusions.

Nobody can find the data that is definitely there

Existing in a Glue database is not the same as being published to the catalog. If an asset was never published, it will not appear in search, and no amount of IAM permission will make it discoverable. Check publication state before you check permissions.

Common mistakes

• Treating it as a migration rather than an addition. Nothing is decommissioned. Your Glue jobs, Athena workgroups and Redshift clusters stay where they were.
• Letting users self-serve projects on day one. Every project is a stack deploy and a set of billable resources. Gate it until profiles and tagging are settled.
• Choosing the domain authentication type in a hurry. It is the hardest thing here to change later.
• Skipping the catalog. Grant everything through direct IAM attachments and you carry the governance overhead with none of the benefit.
• Assuming the CI/CD CLI provisions your platform. It deploys applications into projects that already exist.
• Not tagging project resources from the start. Retrofitting attribution across dozens of blueprint-created resources is miserable work.

How I’d decide

A rough order of operations for anyone weighing this up:

1. Ask what problem you are solving. If it is “our teams cannot discover or safely share each other’s data,” this is the right tool. If it is “we have too many consoles,” it is an expensive way to buy fewer bookmarks.
2. Count your teams. The value curve is steep with cross-team sharing and nearly flat for one team in its own account. One team, one account, mostly model training? Stay on SageMaker AI.
3. Decide the authentication model first. Identity Center for per-user audit trails and approval workflows. IAM if you are optimising for a small group moving fast and accept a different feature set.
4. Write the project profiles yourself. The defaults are demonstrations. Production profiles should encode your networking, tags and quotas.
5. Prove the teardown path before the build path. Create a project, use it, delete it, confirm nothing is orphaned. Do this before any real team touches it.
6. Keep the old consoles. They are your escape hatch when the abstraction hides something you need to see, and cost nothing to leave enabled.

One more piece of judgement: wire your existing observability in from the start rather than relying on the studio’s own views. CloudWatch, Amazon Managed Grafana or a Grafana Cloud stack you already run, it does not much matter which. The failures that matter here are stack failures and job failures, and both are visible in the places you already watch.

Frequently asked questions

Does SageMaker Unified Studio replace SageMaker Studio?

It supersedes it as the recommended experience, and AWS provides an upgrade path that imports existing domains, user profiles and spaces without redeploying infrastructure. JupyterLab and Code Editor spaces, training jobs, pipelines, models and endpoints are preserved, and administrators control when access to the older experience is turned off. It is a migration you schedule, not one that happens to you.

Do I still need Amazon DataZone separately?

Not as a separate purchase. The catalog is built on DataZone, and existing DataZone domains can be upgraded into unified domains, which requires establishing the domain execution and service roles the new model expects. Functionally you are already running DataZone; the studio is the interface on top of it.

How much does SageMaker Unified Studio cost?

The interface has no standalone charge. You pay for each AWS service you use through it on that service’s normal terms, plus SageMaker Catalog usage, metered on metadata ingestion and related operations. Model it by listing what a project profile actually provisions and applying current rates to each item, rather than hunting for a per-seat number that does not exist.

Can I manage domains and projects with Terraform?

AWS’s own documentation directs you to CloudFormation, CDK or Terraform for creating the domain and project substrate, since the CI/CD CLI deliberately does not do it. Coverage in each tool tracks the underlying API, so check the current provider or construct documentation for the specific resources you need before designing around it.

What happens to my Glue jobs and Athena queries?

Nothing. They keep running as Glue jobs and Athena queries. The studio gives you a different place to author and run them and a governance layer that decides who may do so. The execution, the quotas and the billing are unchanged.

Is it worth adopting for a small team?

Often not. The governance machinery is the product, and a team of three sharing one account has little to govern. The threshold is roughly where you have more than one team, data one team owns that another needs, and a real answer required to who approved that access.

Can I run both the old and new experiences at once?

Yes, and you should during transition. The upgrade path lets teams keep access to both while they adapt, with the administrator deciding when to close the older door. Running in parallel for a full billing cycle is also the cleanest way to see what the change actually costs.

The one thing worth remembering

SageMaker Unified Studio replaces an interface problem. It does not replace an infrastructure problem, a permissions problem, or a cost attribution problem, and it adds a governance system you are now responsible for operating.

That trade is worth it when you have data crossing team boundaries and no credible answer to who approved access to it. It is not worth it when the pitch was “fewer tabs.” The teams who are happy a year in went in knowing they were buying a governance engine and budgeted the operational time to run it. The unhappy ones expected a consolidation that was never on offer.

Ask which one you are before the first project gets created, not after the fortieth.

Need a second opinion on your adoption plan?

This is a decision that is cheap to get right at the start and expensive to unwind two quarters later. Things I can help with directly:

• Reviewing a domain design before rollout, especially the authentication model choice and how it meets your existing identity setup
• Writing production project profiles and blueprints that encode your VPC layout, tagging standard and quotas instead of the defaults
• Building the cost attribution layer so project-created resources land in reporting with tags finance can actually use
• Wiring the CI/CD CLI into an existing GitHub Actions, GitLab CI or Jenkins pipeline, plus the CloudFormation or Terraform layer underneath that the CLI does not cover
• Diagnosing failed project creations and access denials that trace back to project roles or Lake Formation grants
• Sanity-checking a migration plan from SageMaker Studio or an existing DataZone domain, including what runs in parallel and for how long

Send me a project profile, a failed stack event log, or a tagged cost export and I will tell you what I see.

The post SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn’t appeared first on John Nessime.

That is the failure worth writing about. Drift itself is normal. Somebody fixes production at 2am, a managed service rotates a value, a policy engine stamps a tag. The part that breaks is the alert, and it breaks quietly, because a muted channel looks identical to a quiet one.

Switching on Terraform drift detection takes one flag. Building Terraform drift detection that engineers still read six months later means deciding what not to send: which plan mode you run, what you classify as noise, how you route by blast radius, and whether the message names who made the change. A detector firing daily with the same content is worse than none, because no detector leaves you knowingly blind while a muted one leaves you blind with a green dashboard. Every section below asks the same question differently: does this message change what a human does in the next ten minutes?

Pick the right plan mode or you will alert on your own backlog

This is the biggest source of false drift, and it catches almost everyone building their first check. A normal terraform plan compares three things: your configuration, your state, and the real infrastructure. It returns a non-zero exit code when any of them disagree, so it fires for real drift and just as loudly for a pull request that merged this morning and has not been applied. Your drift channel becomes a notification feed for your own deployment backlog, and people learn that most messages are work in progress.

Refresh-only mode narrows the question. HashiCorp documents it as a plan whose goal is only to update state and root module outputs to match changes made to remote objects outside of Terraform. That is the actual definition of drift.

terraform init -input=false

terraform plan -refresh-only -detailed-exitcode -lock=false -input=false -no-color

# exit 0  state matches the real infrastructure
# exit 1  the check itself failed
# exit 2  something changed outside Terraform

Two flags there matter more than they look. -detailed-exitcode is what makes the result machine-readable at all: without it a successful plan returns zero whether or not anything changed, so your job passes forever. And -lock=false is safe here specifically because a plan never writes state. Only terraform apply -refresh-only updates the state file. A read-only detector that grabs the state lock will collide with real applies, and the loser of that race is the engineer trying to ship.

The trade-off, stated plainly: refresh-only will not tell you that merged code has not shipped, and it will not find resources created outside Terraform, because it can only refresh what is already in state. I run both modes and route them separately. Refresh-only feeds the alert. Normal plan feeds a weekly digest tied to the repository, where unapplied changes are a delivery problem rather than a security one.

Classify the noise before you route anything

Most recurring drift falls into a few families. Sorting yours takes an hour and decides most of your routing rules.

Provider normalization

A JSON policy document comes back with keys reordered. An identifier returns in different case. An empty block reads back as null instead of an empty map. Terraform sees a difference because it compares a stored representation against an API response, and that round trip is not always lossless. Fix these in the configuration where you can, because a phantom diff that follows you into every plan also hides real diffs on the same resource.

Attributes another system legitimately owns

The classic is autoscaling capacity. Your config says two instances, a scaling policy moved it to five during a spike, and the next plan proposes taking it back to two. Applying that kills three instances mid-spike. The drift is real; reverting it is the wrong move.

resource "aws_autoscaling_group" "web" {
  min_size         = 2
  max_size         = 10
  desired_capacity = 2

  lifecycle {
    # Owned by the scaling policy after creation.
    # Reverting this during a spike removes live capacity.
    ignore_changes = [desired_capacity]
  }
}

Understand what you bought. ignore_changes stops Terraform proposing an update for that attribute, and it applies only after creation, not on the first apply. It also delegates the attribute permanently. If someone sets desired capacity to zero at 3am, your check will not raise a hand. Fair trade for capacity. Terrible trade for an ingress rule.

Tags stamped by something else

Organization tag policies, cost allocation tooling, backup schedulers and compliance agents all add tags. The AWS provider exposes both the tags you set and the effective set including defaults, and ignoring the effective set silences all tag drift at once. Convenient, and slightly dangerous if you use tags for access control, because tag-based policy conditions stop being watched. The same logic covers values the platform rotates or defaults on its own, like a managed secret version: they drift on a schedule that has nothing to do with your team, and belong in a digest.

The goal here is not a clean plan. A clean plan is a side effect. The goal is a decision per family: see it immediately, see it weekly, or never see it again.

Route by blast radius, not by count

“Drift detected: 12 resources changed” makes a human open a terminal before they know whether to care, and that friction is where attention goes to die. Split the output by what each resource can do if it is wrong:

• Wake someone. Anything governing exposure or access: security groups and firewall rules, IAM policies and roles, key policies, public access blocks, bucket policies, DNS records pointing at production.
• Review today. Instance sizes, scaling bounds, database parameters, retention settings, anything that costs money or changes capacity.
• Weekly digest. Tags, descriptions, names, and every family you classified as platform noise.

To split them you need structure, not text. Terraform’s machine-readable output emits a message for each resource that changed outside Terraform, carrying the resource address and the action it would take.

terraform plan -refresh-only -no-color -json 
  | jq -c 'select(.type == "resource_drift") | .change'

Check the exact field names inside that object against the machine-readable output reference for your Terraform version before building matching logic on them. Keep the severity mapping in the repository next to the code it describes. A resource type moving from digest to page should be a pull request, not something changed in a monitoring UI at midnight.

Make the alert answer “who did this”

Terraform tells you what changed. It has no idea who changed it, and that gap is where the investigation time goes. Half the drift you find is a legitimate incident fix nobody wrote down. If the alert names the actor, the response is a two-word reply in the thread instead of twenty minutes of archaeology. On AWS the audit trail can be queried by resource:

aws cloudtrail lookup-events 
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0123456789abcdef0

CloudTrail’s console event history only reaches back a limited window, so a weekly check may be hunting an event that has aged out; a trail delivering to S3, or CloudTrail Lake, covers the gap. Azure exposes the equivalent through Activity Logs, Google Cloud through Cloud Audit Logs. Wiring the lookup into the alert body does more for adoption than any amount of formatting.

Decide the remediation path before you automate it

There are three answers to a drift finding, and picking the wrong one can be worse than the drift.

1. Revert. Apply the configuration and push reality back. Right when the change was unauthorized or wrong.
2. Adopt. Change the code to match what is running. Right when the change was correct and the code was stale, which happens more than people admit.
3. Tolerate. Suppress the attribute deliberately, with a comment and an owner. Right when another system owns it.

The trap is automating option one. Auto-approving a remediation apply sounds like the natural endpoint of a drift pipeline, and it is fine right up until the drifted attribute is one that forces replacement. Then “restore the desired state” means destroying and recreating a live resource, unattended, at whatever hour the schedule runs. If you automate anyway, gate it hard: an allowlist of resource types, a check that the planned action is an in-place update rather than a replace, and a notification either way.

Troubleshooting a drift job that quietly stopped working

The nastiest failure mode is not a noisy check. It is a check reporting “no drift” because it never ran the plan successfully.

Exit code 2 kills your script

With set -e, a non-zero exit terminates the script. Exit code 2 means drift, which is the case you care about, so a naive script dies exactly when it has something to report. Capture the code without letting the shell act on it:

set -euo pipefail

terraform plan -refresh-only -detailed-exitcode -lock=false 
  -input=false -no-color > plan.txt 2>&1 && code=0 || code=$?

case "$code" in
  0) echo "no drift" ;;
  2) notify_drift plan.txt ;;
  *) notify_broken plan.txt; exit 1 ;;
esac

Note that the failure branch notifies too. Exit code 1 deserves the same urgency as drift, because a broken detector and a clean estate produce identical silence.

Credentials that rot without complaining

Drift checks run unattended, so nobody notices when a role trust policy changes, an OIDC subject claim stops matching after a repository rename, or a long-lived key gets rotated out. Add a deadman check: if the job has not reported within its expected interval, that absence is the alert. Same reasoning as heartbeat monitoring for backups.

A provider upgrade that drifts everything

Bump a provider and a new attribute default can appear across every resource of a type. Overnight the check reports hundreds of changed resources and whoever is on call assumes the tool is broken, which it sort of is. Commit the dependency lock file, pin the provider version, run the drift job on the same Terraform version as your apply pipeline, and treat provider upgrades as a reviewed change rather than something a bot merges.

Refresh times that outgrow the schedule

Refresh queries the API for every resource in state. Large workspaces get slow, then overlap their own schedule, then time out. The durable fix is splitting state along boundaries that already exist in your team structure. Terraform does offer targeting to narrow a plan, but it is documented as an exceptional-use option and prints a warning for good reason, so treat it as a stopgap rather than the design.

When to stop building this and buy it

A scheduled job in GitHub Actions or GitLab CI, or a systemd timer on a small always-on box from a provider like Contabo or InterServer, covers a handful of workspaces perfectly well and costs almost nothing. I would not reach for a platform before that stops working. What eventually stops working is the surrounding machinery: credentials for a dozen accounts across providers, concurrency against real applies, and retaining history so you can see that the same resource has drifted repeatedly this quarter, which is a process problem no single alert reveals.

That is where HCP Terraform’s health assessments, or a platform like Spacelift, Scalr or env0, starts earning its fee, since drift detection becomes a workspace setting with history and notifications attached rather than code you maintain. Two honest caveats: HCP Terraform gates health assessments behind its paid editions and requires remote or agent execution, so it is not a drop-in for local-state workflows, and any platform will happily send you the same undifferentiated wall of drift the homegrown script did. Severity tiering is a decision about your infrastructure that no vendor can make for you. Buying the runner does not buy the judgement.

Common mistakes

• Using normal plan mode for the alert. You notify yourself about unapplied work and teach people the channel is noise.
• Alerting on a count instead of a resource. “Nine resources changed” makes someone open a terminal to find out if they should care.
• Treating exit code 1 as a non-event. A detector that errors is indistinguishable from an estate with no drift.
• Taking the state lock for a read-only check. Monitoring should not be able to block a deployment.
• Auto-approving remediation across all resource types. One force-replacement attribute and the fix becomes the outage.

Best practices worth the effort

• Run refresh-only for drift and normal plan for unapplied changes, routed to different destinations.
• Keep the severity mapping in the repository, version controlled and reviewed like code.
• Give every suppression a comment naming the system that owns the attribute and a date to revisit.
• Alert on the detector failing and on the detector going quiet, not only on findings.
• Review recurring drift monthly. The same resource drifting repeatedly is a process gap, not an infrastructure one.

Terraform drift detection FAQ

What is the difference between a normal plan and refresh-only?

A normal plan compares configuration, state and reality, so it flags genuine drift and any committed change that has not been applied. Refresh-only compares state against reality only, which matches what most people mean by drift. Use refresh-only for the alert and normal plan for a delivery-focused report.

Does a refresh-only plan modify my state file?

No. Plan operations do not write state. Only applying a refresh-only plan updates the state file to match the drifted remote objects, and even then it makes no changes to infrastructure. That is why a detection job can safely skip the state lock.

How often should Terraform drift detection run?

Tie the interval to how long you would tolerate not knowing. Daily for production and anything holding customer data, weekly for lower environments, less often for slow-moving workspaces where refresh is expensive. Frequency matters less than whether the message gets read.

How do I find resources created outside Terraform?

Not with a plan. Terraform refreshes only what is already in state, so a resource it has never seen is invisible. You need a cloud-side inventory such as AWS Config, Azure Resource Graph or Cloud Asset Inventory, compared against your state, and import blocks to bring the results under management. Worth knowing that driftctl, the open-source tool most often recommended here, has moved into maintenance mode.

Does ignore_changes hide drift I actually care about?

Yes, and that is the point of using it. It stops Terraform proposing updates for that attribute regardless of what changed it, so a deliberate suppression and a malicious change look identical afterwards. Scope it to specific attributes, never whole resources you still care about, and document the owner.

Why did drift suddenly appear across hundreds of resources?

Almost always a provider or Terraform version change rather than something happening in your cloud account. A new provider release can add or reshape an attribute and every resource of that type reports a difference at once. Compare the versions used by the drift runner and the apply pipeline before investigating the infrastructure.

The one thing worth keeping

Terraform drift detection is not a monitoring problem, it is an attention problem. The flag that finds drift took an afternoon. Everything determining whether anyone still reads the output next quarter is judgement: refresh-only rather than normal plan, noise families sorted and suppressed deliberately, severity by blast radius rather than resource count, an actor name in the message, and an alert when the detector itself goes dark.

Judge it by one test: when a message lands in that channel, does anyone open it? If the honest answer is no, your detection is working and your alert is not.

Need help making your drift alerts worth reading?

This is common work once a Terraform estate grows past one team. Things I can help with:

• Auditing an existing drift job and finding the false positives that trained everyone to ignore it
• Building scheduled refresh-only checks in GitHub Actions, GitLab CI or a systemd timer, with correct exit code handling and a deadman alert
• Sorting recurring drift into families and deciding, per attribute, what to fix, adopt or suppress
• Building severity routing from the plan’s structured output so exposure changes page and tag changes do not
• Enriching alerts with CloudTrail, Activity Log or Cloud Audit Log lookups so findings arrive with an actor attached
• Splitting oversized state so refresh finishes inside its schedule, and judging whether a managed platform is worth it at your size

If you want a second opinion, send me a week of your drift output or the workflow file that produces it, and I will tell you what I would cut first.

The post Terraform Drift Detection That Nobody Mutes appeared first on John Nessime.

That is the real problem with trying to bootstrap a VPS with Ansible in a single run. The failures that cost you are not the ones that turn the recap red. They are the tasks that report changed, do nothing useful, and leave you believing the server is hardened when it is not.

This post covers the ordering that keeps you connected while you dismantle your own access, the silent failure modes that only surface on reboot, and how to make the second run of the playbook as safe as the first. The examples are Debian and Ubuntu flavoured with notes for RHEL-family images, and they assume a plain provider VPS from somewhere like InterServer, Hetzner, DigitalOcean or Vultr, not a pre-baked golden image.

What a one-run bootstrap actually has to survive

A bootstrap playbook is unusual because it modifies the thing it is standing on. Every other playbook you write assumes a working connection. This one has to change the user it connects as, the port it connects to, and the firewall that lets it connect at all, without dropping the run.

Four things break it, and they break it in different ways:

• The chicken-and-egg problem. Ansible needs Python on the target to run almost any module, and a minimal image may not have it.
• Lockout. You remove your own access before proving the replacement access works.
• Non-idempotent tasks. The first run is fine. The second run is where the damage happens.
• Missing collections. The playbook that worked on your laptop fails on the CI runner because half the modules do not ship with ansible-core.

Everything below is organised around those four, because that is how they actually show up.

The chicken-and-egg problem: no Python, no modules

Most modern cloud images ship Python 3 and this never comes up. Minimal Debian netinstall images and some stripped provider templates do not, and then your very first task fails with a message about the interpreter before it has done anything at all.

The escape hatch is ansible.builtin.raw, which pipes a command straight down the SSH connection without needing Python on the other end. Pair it with gather_facts: false, because fact gathering is itself a Python module and will fail before your bootstrap task ever runs.

- name: Make the host manageable
  hosts: new_servers
  gather_facts: false
  become: true
  tasks:
    - name: Check whether a Python interpreter exists
      ansible.builtin.raw: test -e /usr/bin/python3
      register: python_present
      changed_when: false
      failed_when: false

    - name: Install Python if it is missing
      ansible.builtin.raw: apt-get update && apt-get install -y python3
      when: python_present.rc != 0
      changed_when: true

    - name: Gather facts now that we can
      ansible.builtin.setup:

Two details worth understanding rather than copying. failed_when: false on the check turns a non-zero exit into data instead of a failure, so you can branch on it. And ansible.builtin.setup is the fact-gathering module called explicitly, which means from that point on ansible_distribution and friends are available even though the play started with gathering disabled.

If you are provisioning through a provider that supports user-data, doing this in cloud-init instead is cleaner. Let cloud-init handle first-boot truths, then let Ansible own everything that will change later.

Failure family one: locking yourself out while the playbook says ok

This is the one that actually costs a rebuild, and it is entirely an ordering problem. The tasks are all correct in isolation. Run them in the wrong sequence and the playbook cheerfully removes your way back in before it has confirmed the replacement works.

The order that survives

The rule is simple: nothing that removes access happens until something has proven the new access works. In practice:

1. Connect as whatever the provider handed you, usually root or a sudo-capable default user.
2. Create the admin account, add it to the sudo or wheel group, install its authorized key.
3. Write the sudoers drop-in and validate it with visudo -cf.
4. Prove you can log in as that account and escalate. This is the step everyone skips.
5. Install the firewall and allow SSH before enabling it.
6. Write the sshd configuration, validate it, restart, reconnect on the new port and confirm.
7. Only now disable root login and password authentication.

Step four is a real task, not a comment. It runs from the control node against the target, using the credentials you are about to depend on:

- name: Prove the new admin account can log in and escalate
  ansible.builtin.command: id -u
  become: true
  changed_when: false
  vars:
    ansible_user: "{{ admin_user }}"
    ansible_ssh_private_key_file: "{{ admin_key_path }}"

Setting connection variables at the task level means this single task uses the new identity while the rest of the play carries on as root. If the key is wrong or sudo is misconfigured, the play stops here with root login still intact, which is exactly what you want.

The socket activation trap

This is the failure that passes green and bites on reboot. On Ubuntu images from 22.10 onward, sshd is started through systemd socket activation rather than as a persistent service. ssh.socket owns the listening port, and Port in sshd_config is not what decides where the daemon listens.

So your playbook writes Port 2222, restarts the service, reports changed, and the box carries on listening on 22. Your existing session still works. Your firewall rule for 2222 looks fine. Nothing indicates a problem until something forces a reconnect.

The fix is to write a socket override as well, and to check which mechanism the image actually uses rather than assuming:

# Is the socket unit in play on this host?
systemctl is-enabled ssh.socket

# What is actually listening, and under which unit?
ss -tlnp | grep -E 'ssh|:22'

If ssh.socket is enabled, the override needs an empty ListenStream= first to clear the inherited default, then the real value. Miss the empty line and the host listens on both ports, which is worse than either outcome alone because your hardening looks applied and is not.

Honestly, the simpler answer for most single-server setups is to leave SSH on 22 and lean on key-only authentication plus a firewall source restriction. Moving the port cuts log noise from opportunistic scanners. It does not stop anyone who is actually looking at you.

Firewall ordering

UFW defaults to denying inbound traffic when you enable it. Enable first, add the SSH rule second, and the run dies mid-task with an UNREACHABLE that will not recover. Rules go in first, always:

- name: Allow SSH before the firewall is armed
  community.general.ufw:
    rule: allow
    port: "{{ ssh_port }}"
    proto: tcp

- name: Set default inbound policy
  community.general.ufw:
    direction: incoming
    policy: deny

- name: Enable UFW
  community.general.ufw:
    state: enabled

Two extra things bite here. Provider-level firewalls, the ones configured in the control panel rather than on the host, are a second layer that Ansible has no visibility into. If you open a port on the host and traffic still does not arrive, check there before debugging iptables. And if you moved the SSH port, the host firewall rule and the provider firewall rule both need updating, in that order, before the daemon restarts.

Failure family two: the second run is the dangerous one

A bootstrap playbook you run once is a shell script with extra syntax. The value only appears when you can run it again on the same host, in six months, and have it either do nothing or fix drift. That is idempotency, and the usual killers are shell and command tasks with no guard.

An unguarded shell task reports changed every single time, which means any handler it notifies fires every single time. That is how a playbook you thought was a no-op ends up restarting sshd on a production box during a routine drift check.

Guard them properly:

• creates: or removes: makes the task skip when the target state already exists. Prefer this.
• changed_when: false for read-only commands so they stop lying about changing things.
• when: driven by a registered check, for anything conditional.

The other idempotency trap is lineinfile against sshd_config. It works, and it is fragile in a specific way: the regex has to match the shipped file, and different distributions ship different defaults, commented or not. Writing a drop-in file is more predictable:

- name: Write the sshd hardening drop-in
  ansible.builtin.template:
    src: 99-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/99-hardening.conf
    owner: root
    group: root
    mode: "0600"
  notify: Restart sshd

- name: Validate the full sshd configuration
  ansible.builtin.command: sshd -t
  changed_when: false

Note that the validation runs against the whole configuration rather than the drop-in in isolation. A drop-in is not a valid standalone config file, so validate: on the template task with sshd -t -f %s will fail on a temporary file that has no host keys and no context. Run sshd -t as its own task afterwards instead, and let the handler restart only if it passes.

Two prerequisites for the drop-in approach. The base sshd_config needs an Include /etc/ssh/sshd_config.d/*.conf line, which recent Debian, Ubuntu and RHEL 9 images have but older ones may not. And in sshd configuration the first value for a keyword wins, so the include has to appear near the top of the file for your drop-in to override anything below it. Check both before relying on the mechanism.

Failure family three: the collections are not where you think

ansible-core ships the ansible.builtin namespace and essentially nothing else. The full ansible package bundles a large set of community collections, which is why a playbook that runs on your laptop can fail on a lean CI runner with a module-not-found error.

Things people assume are built in and are not: ufw and timezone live in community.general. authorized_key, sysctl and mount live in ansible.posix. Declare them and pin them:

# requirements.yml
collections:
  - name: community.general
  - name: ansible.posix

# install before the run, and in CI
ansible-galaxy collection install -r requirements.yml

Use fully qualified names everywhere. Short names still resolve, but they resolve through a lookup that can pick a different module when two collections define the same name. ansible-lint flags this and it is worth listening to.

Failure family four: secrets in the repository

Bootstrap playbooks accumulate secrets faster than anything else you write: initial passwords, API tokens for monitoring agents, TLS material, backup credentials. They also tend to be the first repository someone shares with a client.

Ansible Vault is the low-friction option and it is genuinely fine for a small estate. Encrypt the variable file, not the playbook, so diffs stay readable:

ansible-vault encrypt group_vars/all/vault.yml
ansible-playbook site.yml --ask-vault-pass

The honest trade-off: Vault gives you one shared passphrase with no rotation story and no audit trail. Once more than two people run the playbook, or a client needs their own credentials kept separate from yours, move to something with real access control. That is more moving parts, so do not reach for it on day one.

Putting it together: how to bootstrap a VPS with Ansible end to end

Structure matters less than order, but a shape that works well is a single site.yml calling roles in dependency order, with the risky access changes fenced off at the end.

# inventory.yml
new_servers:
  hosts:
    web01:
      ansible_host: 203.0.113.10
      ansible_user: root
      ansible_port: 22
  vars:
    admin_user: deploy
    ssh_port: 22

Then the run itself, in this order:

1. Reachability. Python check, fact gathering, timezone, hostname, package cache refresh.
2. Identity. Admin user, group membership, authorized key, sudoers drop-in with visudo -cf validation.
3. Verification gate. The task shown earlier. Nothing destructive runs before this passes.
4. Firewall. Rules first, policy second, enable third.
5. Unattended updates. unattended-upgrades on Debian and Ubuntu, dnf-automatic with its systemd timer on RHEL-family hosts.
6. Brute-force protection. fail2ban with your own /etc/fail2ban/jail.local, never edits to jail.conf.
7. Workload. Docker, a web server, a monitoring agent, whatever the box is for.
8. Access teardown. Disable root SSH login, disable password authentication, restart sshd, reconnect, confirm.

When step eight changes the port or user, the play needs to actually reconnect rather than reusing its cached SSH connection. Ansible keeps a persistent connection per host, so a config change alone will not be picked up:

- name: Point subsequent tasks at the new identity
  ansible.builtin.set_fact:
    ansible_user: "{{ admin_user }}"
    ansible_port: "{{ ssh_port }}"

- name: Drop the cached connection
  ansible.builtin.meta: reset_connection

- name: Confirm the host answers on the new settings
  ansible.builtin.wait_for_connection:
    timeout: 60

meta: reset_connection does not accept a when conditional, so it runs on every pass. That is a small cost for a play that works on the first run and every run after.

Test it before you point it at anything real

The cheapest test is a disposable VPS. Spin one up, run the playbook, reboot it, run the playbook again, then destroy it. The reboot is the important part, because that is what exposes the socket activation problem and any service you enabled but never actually started.

# catch YAML and structural errors instantly
ansible-playbook site.yml --syntax-check

# style and correctness, including FQCN and idempotency smells
ansible-lint

# show what would change without changing it
ansible-playbook -i inventory.yml site.yml --check --diff

One caveat on --check: it is not fully reliable on a bootstrap playbook. Tasks that depend on earlier tasks having run will fail or report nonsense in check mode, because the earlier change did not really happen. Treat a clean check run as a weak signal, not proof.

Troubleshooting

• UNREACHABLE on the very first task. Usually host key verification against a rebuilt host reusing an old IP. Remove the stale entry with ssh-keygen -R and re-add the correct one with ssh-keyscan. Disabling host key checking globally makes this go away and removes a real protection, so do not.
• UNREACHABLE immediately after a firewall task. You armed the firewall before allowing SSH. Recover through the provider console, not SSH.
• “Missing sudo password”. The sudoers drop-in is not in effect or has bad syntax. Validate with visudo -cf on the file itself before trusting it.
• Module not found under an FQCN. The collection is not installed on the machine running the playbook. Confirm with ansible-galaxy collection list.
• New group membership ignored. Adding a user to docker mid-play does not affect the already-open connection. Insert meta: reset_connection after the group change.
• fail2ban installed but not banning. On images without rsyslog, /var/log/auth.log may not exist, so a jail configured to read it never starts. Check fail2ban-client status and switch the jail to the systemd backend if the log file is absent.
• Everything green, port unchanged. Socket activation. Check systemctl is-enabled ssh.socket and ss -tlnp.

Common mistakes

• Disabling root login in the same play that creates the replacement user, with no verification step between them.
• Enabling UFW before the SSH allow rule exists.
• Assuming Port in sshd_config is authoritative on every distribution.
• Unguarded shell tasks that notify handlers, causing service restarts on every run.
• Editing jail.conf instead of creating jail.local, so the next package update silently reverts it.
• Committing plaintext secrets because “it is only a bootstrap playbook”.
• Testing on a host you cannot afford to lose, with no console access configured.
• Relying on --check as a substitute for a real run on a disposable host.

Best practices

• Know your recovery path before the first run. Provider console, serial console or recovery mode, tested once so you are not learning it under pressure.
• Put a verification task before every irreversible one, and let the play fail there rather than after.
• Prefer drop-in files over line edits for sshd, sudoers, fail2ban and sysctl. They survive package upgrades and are trivially removable.
• Validate configuration before restarting the service that reads it. sshd -t, visudo -cf, nginx -t.
• Pin collections in requirements.yml and install them in CI, not just locally.
• Run the playbook twice on a fresh host and read the second recap. Anything still reporting changed is a bug.
• Reboot once during testing. Half of the silent failures only appear after a restart.
• Get a monitoring agent on the box in the same run. A server that is hardened but unobserved is a server you will hear about from a customer first.

Frequently asked questions

Can you really bootstrap a VPS with Ansible in a single playbook run?

Yes, including changing the SSH user and port mid-run. The mechanism is set_fact to update the connection variables, meta: reset_connection to drop the cached connection, and wait_for_connection to confirm the new settings work before continuing. What you cannot do safely is change access and skip the confirmation.

Should I use cloud-init or Ansible for initial server setup?

Both, split by lifetime. cloud-init handles things that are true exactly once at first boot: the admin user, the SSH key, a minimal package set. Ansible handles everything that will be tuned again later. Putting hardening rules in cloud-init means the only way to change them is to rebuild the host.

Do I need Terraform or OpenTofu as well?

Only if you are creating the infrastructure, not just configuring it. Terraform and OpenTofu create the VPS, the DNS records and the provider firewall. Ansible configures what runs inside. For a handful of long-lived servers ordered through a provider panel, Ansible alone is enough and adding a second tool costs more than it returns.

Why does my playbook work the first time and fail the second?

Almost always because the first run changed the connection parameters and the inventory still describes the original ones. Either update the inventory to the post-bootstrap values and make the play tolerate both, or split bootstrap and steady-state into separate playbooks with separate inventory entries.

Is changing the SSH port worth doing?

It reduces log noise from untargeted scanning, which makes real events easier to spot. It is not a security control on its own. Key-only authentication, a default-deny firewall and fail2ban do the actual work. If moving the port complicates your automation, the honest answer is to leave it on 22.

How do I recover if the playbook locks me out?

Through the provider’s console or recovery environment, which is why you should know where it is before you need it. Most VPS providers offer a web-based console, some offer a rescue boot with the disk mounted. From there, revert the offending drop-in file or firewall rule, restart the service, and reconnect. Then add the verification step the playbook was missing.

Can this run against several servers at once?

Yes, but use serial: 1 the first few times. If there is an ordering bug in the play, running it against one host at a time means you lock yourself out of one server instead of a fleet.

The one thing worth remembering

When you bootstrap a VPS with Ansible, the individual tasks are the easy part. Creating a user, writing a firewall rule and hardening sshd are all well-documented and hard to get wrong in isolation. What decides whether the run leaves you with a working server or a rebuild is the sequence, and specifically whether you verify new access before removing old access.

Add the verification gate. Run the playbook twice. Reboot the box once. Those three habits catch nearly every silent failure described here, and they cost about ten minutes on a disposable host.

Need a bootstrap playbook you can actually rerun?

Most of the provisioning work I take on starts with a playbook that worked once and has been too frightening to run since. Things I help with:

• Building a bootstrap role that takes a blank VPS to a hardened, monitored server in one run, with a verification gate before every irreversible step.
• Auditing an existing playbook for idempotency, so the second and hundredth runs are as safe as the first.
• Untangling SSH, firewall and socket activation interactions on mixed Debian, Ubuntu and RHEL-family estates.
• Splitting cloud-init and Ansible responsibilities cleanly so hardening stays changeable after first boot.
• Moving secrets out of a repository into Vault or a managed secret store without breaking existing runs.
• Wiring the playbook into CI with linting, collection pinning and a disposable-host test that runs on every change.

Send me the playbook, a recap from a second run, or the output of ss -tlnp from the host that is behaving oddly, and I will tell you what I would change first.

The post Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out appeared first on John Nessime.

Nothing about the database had changed. Same engine, same instance class, same subnet group. What changed was the resource’s name inside the module. The author had tidied up a resource "aws_db_instance" "this" into something more descriptive, shipped it as a minor release, and moved on. Terraform saw a resource at an address that no longer existed and a resource at an address that did not exist before. Its default reading of that is destroy and create.

That is the real cost of a module, and it is invisible on day one. The first apply is always fine. Modules feel great the week you adopt them. The bill arrives at the third upgrade, when the person who wrote it has moved teams and a dozen repositories consume it.

This post is about judging that cost before you pay it. What separates reusable Terraform modules that are genuinely worth adopting from the ones your team quietly pins at an old version and refuses to touch, how to make your own modules land on the right side of that line, and what to do when a plan comes back looking like the one above.

The bill for a module arrives at upgrade time

Terraform correlates state to configuration by address. Not by tags, not by cloud resource ID, not by anything the provider knows. Just the address: module.database.aws_db_instance.this. Change any part of that path and, unless you say otherwise, Terraform concludes the old object should go and a new one should take its place.

For a security group that is an annoyance. For a database, an object store, or anything holding data, it is an incident with a retro attached.

The fix has existed for a long time and it is genuinely good: the moved block. A module author who renames something declares the rename in code, ships it with the release, and every consumer’s next plan updates state instead of destroying infrastructure.

# Shipped inside the module, in the same release as the rename.
# Terraform rewrites the state address instead of planning a replacement.

moved {
  from = aws_db_instance.this
  to   = aws_db_instance.primary
}

The block is declarative and it chains. If a resource moves in one release and moves again two releases later, someone jumping several versions at once still lands correctly, because Terraform resolves the whole chain. That is why a module that keeps its moved blocks around is doing you a favour, and why one that strips them out after a release or two is quietly making every skipped upgrade riskier.

So here is the whole thesis, and everything below is elaboration: a module is worth reusing if the next version bump is cheap. Everything else is secondary.

Five signals that reusable Terraform modules are worth adopting

These are the things I actually check before letting a module into a codebase. None of them take more than a few minutes. All of them predict the upgrade experience better than star counts do.

1. It keeps state addresses stable, or tells you when it does not

Open the repository. Search for moved. Read the changelog for the last major release and see whether renames are called out.

A module with no moved blocks and no changelog is not necessarily bad code. It just means every internal refactor becomes your problem, discovered in a plan output, usually under time pressure. A module that says “v3.0 renames the launch template resource, here are the moved blocks” is telling you it has thought about consumers.

2. The input surface is narrow and typed

Count the variables. If a module needs forty inputs to cover its use cases, the abstraction was drawn in the wrong place. Usually that means two patterns got merged into one module because they looked similar from a distance.

Width is not the only tell. Look at how the inputs are typed. A module that takes twelve loose strings is very different from one that takes three typed objects with sensible defaults baked in.

variable "logging" {
  description = "Access log settings. Set to null to disable logging."
  type = object({
    bucket         = string
    prefix         = optional(string, "logs/")
    retention_days = optional(number, 30)
  })
  default = null
}

variable "environment" {
  description = "Deployment environment."
  type        = string
  nullable    = false

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be one of dev, staging or prod."
  }
}

Three things are doing work there. optional() with a default means a caller who wants the simple case writes two lines instead of six. nullable = false means a caller cannot pass null and get a confusing failure three resources deep. The validation block means a typo fails at plan time with a sentence a human wrote, rather than as a provider error about an invalid parameter.

That last one matters more than it looks. Every validation rule a module author writes is an error message you do not have to debug.

3. You can pin it, and the pin means something

This trips people up because the two common source types behave differently.

# Registry source. The version argument applies here.
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"   # allows any 5.x, refuses 6.0
}

# Git source. There is no version argument. Pin with ref.
module "internal_network" {
  source = "git::https://git.example.com/infra/tf-network.git?ref=v2.3.1"
}

The ~> operator only allows the rightmost specified component to increase. Written against a major and minor, as above, it accepts patch and minor releases and blocks the next major. Written as ~> 5.0.1 it accepts patches only. Which one you want depends entirely on whether you trust the author’s reading of semantic versioning, and the honest answer for third-party modules is usually “not enough to auto-accept minors in production”.

For Git sources, pin to a tag, not a branch. A branch reference resolves to whatever the tip happens to be at init time, which turns your infrastructure into a moving target and makes a plan from two weeks ago unreproducible.

4. It has tests you can run without a cloud account

Terraform’s built-in test framework uses .tftest.hcl files and the same HCL you already write. The critical detail for module authors is that a run block set to command = plan creates nothing. No credentials, no resources, no teardown, no cost. It just checks that the configuration produces what you claimed.

# tests/validation.tftest.hcl
# Runs on plan only. Nothing is created and nothing is billed.

run "rejects_unknown_environment" {
  command = plan

  variables {
    environment = "produciton"
  }

  expect_failures = [var.environment]
}

Negative tests like that are the cheapest quality signal in the whole ecosystem. They prove the validation rules actually fire, which is the difference between a rule that catches typos and a rule that was written once and silently broken later.

OpenTofu runs the same format through tofu test, and additionally recognises a .tofutest.hcl extension for tests that should only run there. Useful if you maintain modules that target both tools.

5. Somebody answers issues

Least technical, most predictive. Open the issue tracker, sort by recently updated, and look at whether maintainers reply. A module with a hundred open issues and active triage is in far better shape than one with six open issues and no comments since a provider major release.

Provider major versions are the stress test. When the AWS or Azure provider ships a breaking change, every module has to react. Modules without a maintainer simply stop being usable, and you find out when init refuses to resolve a version constraint.

Three kinds of module, and which ones actually pay off

Not everything that can be a module should be. It helps to name the three shapes you see in the wild.

The thin wrapper

One resource, a handful of variables passed straight through, maybe a tag merge. This is the most common module and usually the least valuable. It adds a layer of indirection, a version to track and a state address prefix, in exchange for saving four lines.

There is one case where it earns its place: when the wrapper is enforcing something, like mandatory encryption, a naming convention, or a tag schema your finance team depends on. Then it is not a wrapper, it is a policy with a friendly interface. If it enforces nothing, inline the resource and move on.

The composition module

Five to fifteen resources that only make sense together. A VPC with its subnets, route tables and gateways. A service with its task definition, target group, listener rule and log group. This is where modules earn back their cost, because the value is in the wiring, and the wiring is exactly what people get wrong when they copy and paste.

Most modules worth reusing live here.

The landing zone

An entire environment behind one module call. Accounts, networking, IAM baseline, logging, the lot. These are genuinely powerful and genuinely expensive. They only work when a team owns them, tests them, and has a release process. A landing zone module maintained by one person as a side project becomes the single scariest thing in the repository, because nobody can upgrade it and nobody can replace it either.

Public modules or your own?

The case for public modules is stronger than the “we have specific needs” crowd usually admits. The community AWS collections in particular have absorbed years of edge cases you have not thought of yet: the ordering constraints, the optional sub-resources, the arguments that only apply in certain regions. Reimplementing that from the provider docs is a real project, not an afternoon.

The case against is equally real. Popular public modules are wide by necessity, because they serve everyone. That width shows up as a large input surface, a lot of conditional logic, and majors that shuffle internals. You inherit an upgrade cadence you do not control.

My rough split: use public modules for undifferentiated infrastructure where the community has clearly solved it, and write your own for anything encoding a decision your organisation made. A VPC is undifferentiated. Your service deployment pattern is not.

There is a licensing dimension worth knowing about. Terraform moved from an open source licence to the source-available Business Source License, which is what prompted the OpenTofu fork now under the Linux Foundation. Module code itself is unaffected by that change, and the two tools read the same HCL, so modules generally work with either. OpenTofu runs its own registry rather than HashiCorp’s. If your organisation has strict open source licensing requirements, that distinction is the one to escalate, and it is a legal question rather than a technical one.

For sharing modules internally, a private registry gives you discovery and a version list in one place. HCP Terraform includes one, and the run platforms built around Terraform and OpenTofu, such as Spacelift, Scalr and env0, all offer module catalogs with policy enforcement attached. You can also get most of the benefit from tagged Git repositories on GitHub or GitLab and a README, which is where I would start before paying for anything. If you self-host your Git server and CI runners, they need somewhere to live, and a small VPS from a provider like InterServer, Hetzner or DigitalOcean handles module CI comfortably, since the plans are cheap and the tests never leave the plan phase.

Making your own modules worth reusing

If you are on the authoring side, this is the order I would do things in.

1. Write it inline first, twice. Do not modularise a pattern you have seen once. The second copy tells you which parts vary and which parts are actually fixed. That is the module’s interface, discovered rather than guessed.
2. Extract with moved blocks in the same commit. Moving a resource into a module changes its address. If the resource holds data, that is the difference between a refactor and an outage.
3. Type the inputs properly. Objects with optional() defaults over flat strings, nullable = false where null makes no sense, validation blocks with error messages that name the fix.
4. Prefer for_each over count for anything collection-shaped. With count, removing an element from the middle of a list shifts every index after it, and Terraform plans a rebuild of resources you never touched. Keyed addresses do not shift.
5. Generate the docs. terraform-docs reads your variables and outputs and writes the README table. Wire it into CI so the docs cannot drift from the code, because hand-written input tables are always out of date.
6. Lint and scan on every PR. terraform fmt -check, terraform validate, tflint for provider-specific mistakes, and a policy scanner such as Checkov or Trivy for the security defaults. All of these run in seconds on a plain GitHub Actions or GitLab CI runner.
7. Tag releases, and mean it. A major for anything that changes an address or removes an input. A minor for new optional inputs. Consumers pin against your tags, so the tag is a promise.

One thing I would skip early on: building a full integration test suite that applies real infrastructure. It is slow, it costs money, and it needs credentials in CI. Plan-mode tests plus a policy scanner catch most of what actually breaks. Add applied tests later, for the two or three modules where the wiring is genuinely subtle.

Troubleshooting the plans that scare you

“Forces replacement” after a version bump

Before you argue with it, find out what it wants to delete. Save the plan and read it as JSON rather than scrolling terminal output.

terraform init -upgrade
terraform plan -out=tfplan

# List every address the plan intends to delete.
terraform show -json tfplan | jq -r '.resource_changes[]
  | select(.change.actions[] == "delete") | .address'

If the list is empty, the replacement was an argument change, not an address change, and you should read the diff on that resource. If the list contains things you did not expect, compare the addresses against the new module version’s source. A rename with no moved block is the usual answer, and you can write the moved block yourself in your root configuration as a workaround.

A module with a provider block inside it

Modules should declare which providers they need, not configure them. A module that contains a configured provider block rather than just a required_providers entry limits what callers can do with it, including using for_each or count on the module call. It also makes the module harder to destroy cleanly. If you hit that, the fix is on the module side: move provider configuration up to the root and pass configuration aliases in.

Changes to the module source are not picked up

Remote modules are downloaded into the working directory during init. Changing a ref or a version constraint in your configuration does not re-download anything on its own. Run terraform init -upgrade. If your CI caches the module directory between runs, that cache is the first place to look when a pipeline plans against code you are sure you deleted.

Registry version list looks stale

A registry’s list of versions is a synced view of your Git tags, not an independent database. Reconnecting a VCS integration, moving repositories, or pushing many tags at once can leave the registry out of step with the repository. Treat the tags as the source of truth and check the registry’s view after any change to that integration.

Common mistakes

• Pinning to a branch. ?ref=main is not a pin. It is an unannounced deployment scheduled for whenever someone next runs init.
• Modularising after one use. You end up encoding the first use case’s assumptions as the abstraction, then bolting flags onto it for every case after.
• Copying environments instead of parameterising them. Three near-identical directories drift within a quarter, and the fix that went into dev never reaches prod.
• Passing every provider argument straight through. If a module’s variables are a mirror of the resource schema, it is not an abstraction, it is a rename with extra steps.
• Deleting moved blocks too early. They only cost you a few lines. Removing them is safe only when you are certain every consumer has applied past that version, which you can rarely be sure of for a public module.
• Applying a plan you did not read. Nearly every module horror story ends with someone approving a plan whose destroy count they never looked at.

How I decide whether to adopt a module

Roughly ten minutes, in this order:

1. Count the input variables. Over about thirty, I assume the abstraction is wrong and look for a narrower alternative.
2. Grep for moved. Its presence tells me the author thinks about consumers.
3. Read the last major release notes. Are breaking changes explained, with an upgrade path?
4. Check for a tests directory and whether CI runs on pull requests.
5. Look at the issue tracker’s recent activity, not its total count.
6. Adopt it in one non-production configuration first, then bump it once deliberately and watch the plan. That single bump tells you more than the other five steps combined.

And a standing rule regardless of the module: never let a plan with a non-zero destroy count through review without someone saying out loud what is being destroyed and why.

Frequently asked questions

How many input variables is too many for a Terraform module?

There is no hard limit, but a wide input surface is a symptom rather than a style choice. If covering your use cases needs dozens of inputs, the variation between callers is probably larger than what they share, which means you have found two patterns rather than one. Splitting into two narrower modules usually reads better and upgrades more safely than adding a twelfth boolean flag.

Should I use public modules or write my own?

Both, for different things. Public modules are a good trade for undifferentiated infrastructure where the community has already absorbed the edge cases. Write your own for anything that encodes a decision specific to your organisation, because that is the part nobody else will maintain for you and the part that changes on your schedule rather than someone else’s.

Do Terraform modules work with OpenTofu?

Generally yes. A module is just a directory of HCL, and both tools read the same language and use compatible provider plugins, so registry modules, private modules and Git-sourced modules usually work unchanged. Where you can hit friction is version constraints and features that have diverged between the two projects since the fork. Test one workspace before committing an estate to a migration.

How do I pin a module version from a Git source?

The version argument only applies to registry sources. For Git, append a ref query parameter to the source URL and point it at a tag or a commit SHA. Branch names resolve to whatever the tip is at init time, which defeats the purpose of pinning entirely.

What happens if a module removes a resource in a new version?

Terraform plans to destroy it, because the resource is in state and no longer in configuration. Sometimes that is exactly what the author intended. Sometimes it means functionality moved somewhere you have not adopted yet. Read the release notes before applying, and if the object holds data, take a snapshot first regardless of what the notes say.

Can I test a Terraform module without cloud credentials?

Yes, for a useful subset. Run blocks set to command = plan evaluate the configuration without creating anything, which is enough to assert on computed values, defaults and validation failures. Testing behaviour that only exists after creation still needs an apply and therefore real credentials, which is why most module suites are mostly plan tests with a small number of applied ones.

Do I need a private registry to share modules internally?

No. Tagged Git repositories and a source URL cover the mechanics completely. A registry buys you discovery, a browsable version list and a place to attach policy, which starts mattering once enough teams are publishing that people cannot find what already exists. Below that point it is overhead you do not need yet.

Conclusion

Most advice about reusable Terraform modules focuses on the writing: structure, naming, README quality. That is the easy part, and it is not where things go wrong.

The modules that are actually worth reusing are the ones you can still upgrade in two years. Stable state addresses, a narrow typed interface, a pin that means something, tests that run on plan, and a maintainer who replies. If you only remember one thing, remember what a module is really promising you: not that the first apply will work, but that the next version bump will not wake anyone up.

Need a second pair of eyes on your module library?

I work with teams on the boring, high-consequence parts of Terraform and OpenTofu estates. Things I help with regularly:

• Auditing an existing module library and saying plainly which modules earn their keep and which should be inlined back into root configurations
• Adding moved blocks and upgrade notes to in-house modules so version bumps stop being incidents
• Narrowing wide module interfaces with typed objects, optional() defaults and validation rules that fail with useful messages
• Setting up module CI: fmt, validate, tflint, policy scanning and generated docs on every pull request
• Writing terraform test suites that run on plan, including negative tests that prove your validation rules actually fire
• Planning a Terraform to OpenTofu migration across a module estate, or working out whether it is worth doing at all

Send me a module’s variables.tf and the plan output from your last version bump. That pair is usually enough for me to tell you where the trouble is.

The post Terraform Modules Worth Reusing (And the Ones That Cost You) appeared first on John Nessime.

That is the failure that actually bites when you provision servers by hand for a handful of clients. It is invisible for months. Nothing alerts. Nothing breaks. Then a disk dies, or a client asks for a second region, or you want to hand the whole thing over, and the gap between “the server works” and “the server is reproducible” turns into a weekend.

This post covers how to automate server provisioning for new clients in a way that survives that moment: what to put in the image, what to put in first boot, what to leave to configuration management, how to keep one client’s credentials from ever touching another client’s box, and how to prove a build is finished instead of assuming it.

The four failure families worth designing against

Automation is not the goal. Automation is what you reach for once you have decided which of these you refuse to keep living with. Pick the ones that apply to you and let them shape the build, because trying to solve all four at once is how provisioning projects stall.

1. You cannot rebuild it

The test is blunt: destroy the machine and rebuild it from your repository, with no shell access to the old one. If you cannot do that, you do not have a server, you have an artefact. Every hand-run apt install, every file edited over SSH at 1am, every “I’ll document that later” is a piece of the machine that exists nowhere but the machine.

This is the one that stays invisible longest. A hand-built box can run happily for two years. The cost lands entirely in one bad afternoon.

2. Every client is a slightly different server

Client A got fail2ban. Client B got it two months later with different jail settings. Client C never got it because that build happened on a Friday. Now a CVE lands in a package you installed on four of six boxes and you have to go look at each one to find out.

Drift is not just untidy. It destroys your ability to reason about the fleet as a fleet. Any statement that starts “all our clients have…” becomes a guess.

3. One credential, too much blast radius

This one is specific to consulting and it is the one people get wrong most often. If the same SSH key opens every client’s server, and the same API token provisions every client’s infrastructure, then a compromise of your laptop is a compromise of everybody. You are a shared dependency with none of the controls a shared dependency is supposed to have.

Worse, it is contractually awkward. Plenty of client agreements have language about access control that a single reused key quietly violates.

4. The handover you cannot perform

Engagements end. When one does, the client should be able to take a repository, run it, and get their infrastructure. If the only way to reproduce their environment is to hire you again, that is not leverage, it is a liability. It also makes you slower to say yes to the next client, because your capacity is tied up in machines only you understand.

How to automate server provisioning for new clients

The stack that holds up in practice has four layers, and the important part is the boundary between them. Most broken provisioning setups I see are broken because someone pushed work into the wrong layer.

Layer 1: the machine itself

Terraform or OpenTofu talks to your provider’s API and creates the server, the firewall rules, the volumes and the DNS records. Whether you run a VPS at InterServer, Hetzner, Vultr or DigitalOcean, or full cloud on AWS, the shape is the same: a declarative file that says what should exist.

Worth knowing before you commit: HashiCorp moved Terraform from the MPL to the Business Source License, which is source-available rather than open source, and the community forked the last MPL release as OpenTofu under the Linux Foundation. For an independent engineer building client infrastructure, either is workable. Terraform has the larger provider and module ecosystem and the commercial backing. OpenTofu gives you a licence that will not be renegotiated under you, which matters more when the code you write ends up in a client’s repository and out of your control. The HCL, provider protocol and state model are shared, so the switching cost is low in both directions. I lean OpenTofu for client work for the licence certainty, and that is a judgement call, not a technical verdict.

Keep state remote and locked from day one, and keep one state file per client. Shared state is the fastest way to turn a mistake on one engagement into an outage on another.

Layer 2: first boot with cloud-init

cloud-init runs once, on first boot, before you have any way in. Its job is narrow: create the account you will connect as, install its key, lock down password and root login, and get out of the way. Everything else belongs in the next layer.

#cloud-config
hostname: web01
fqdn: web01.acme.internal
timezone: Etc/UTC

users:
  - name: deploy
    groups: [sudo]
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nz... provisioning key for this client only

# Refuse passwords and direct root logins from the very first boot,
# so there is no window where the box is reachable with weaker settings.
ssh_pwauth: false
disable_root: true

package_update: true
packages:
  - python3
  - sudo

Two deliberate omissions. There is no package_upgrade: true, because a full upgrade at first boot can run for minutes, holds the package lock, and is the single most common cause of the race described in the troubleshooting section below. Do the upgrade in Ansible where you can see it fail. And there is no application setup, because anything in runcmd is a script you will never test again.

python3 is there so Ansible has an interpreter to work with. Some images ship it already; installing it is cheap insurance.

Validate the file before it ever reaches a provider, because a YAML error in user-data fails silently and you find out by not being able to log in:

cloud-init schema --config-file user-data.yaml

Layer 3: convergence with Ansible

This is where the actual build lives: packages, users, firewall, TLS, web server, database, backup agent, monitoring agent, log shipping. Ansible is agentless and runs over SSH, which fits consulting well because you are not asking a client to accept a permanent daemon on their box.

The layout that scales across clients separates the shared build from the per-client differences:

infra/
  roles/                  # shared, versioned, the same for everyone
    base/
    hardening/
    web/
    backup/
  clients/
    acme/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
    globex/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
  site.yml

Roles are the product. Client directories are configuration. If you find yourself copying a role into a client directory and editing it, that is a variable you have not extracted yet.

Hand the inventory over from Terraform rather than maintaining it by hand. The simplest reliable version reads the outputs as JSON:

cd clients/acme/terraform

tofu output -json client_hosts 
  | jq -r '.value[] | "(.name) ansible_host=(.ipv4)"' 
  > ../inventory/hosts.ini

There are richer options, including inventory plugins that read Terraform state directly, but a generated file has a property worth keeping: you can read it, diff it, and commit it. When a provisioning run targets the wrong host, that diff is how you find out.

Layer 4: the verification gate

This is the layer almost everyone skips, and it is the one that turns provisioning from a script into a process. A playbook that finishes without errors has told you that no task raised an exception. It has not told you the server works.

Write a separate play whose only job is to assert the finished state, and run it as the last step of every provision:

- name: Verify the host is genuinely ready
  hosts: all
  gather_facts: true
  tasks:
    # raw, not command: this may run before Python is guaranteed present
    - name: Wait for cloud-init to finish
      ansible.builtin.raw: cloud-init status --wait
      changed_when: false

    - name: Collect service state
      ansible.builtin.service_facts:

    - name: Core services are running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'
          - ansible_facts.services[ssh_unit].state == 'running'
        fail_msg: "Core services are not up on {{ inventory_hostname }}"

    - name: Site answers over TLS
      ansible.builtin.uri:
        url: "https://{{ client_domain }}/"
        status_code: 200
      delegate_to: localhost

Note ssh_unit as a variable rather than a literal: the SSH unit is ssh.service on Debian and Ubuntu but sshd.service on RHEL-family systems. Hard-coding it is how a verification play passes on one distro and fails confusingly on another.

The point of the gate is not that it catches exotic bugs. It is that it catches the ordinary ones: the service that installed but did not enable, the certificate that issued but did not get loaded, the firewall rule that opened the wrong port.

Keeping clients apart: secrets and access

Generate a fresh SSH keypair per client engagement. It costs you thirty seconds and it means revoking access to one client’s infrastructure is a single, contained action.

For secrets in the repository you have two credible options. Ansible Vault is built in and needs nothing extra, but it encrypts whole files, so your diffs become meaningless blobs and rotating a key across several clients is tedious. SOPS encrypts values while leaving keys readable, which means a code review can still see that a variable changed without seeing what it changed to, and it supports age, PGP and the cloud KMS services. For a repo that a client might eventually inherit, that readability is worth the extra tool.

# .sops.yaml at the repo root: different key per client
creation_rules:
  - path_regex: clients/acme/secrets/.*.ya?ml$
    age: age1acme...
  - path_regex: clients/globex/secrets/.*.ya?ml$
    age: age1globex...

Separate keys per client is the whole point. One shared key and you are back to a single blast radius with extra ceremony.

For handing the initial credentials to a client, use something with an audit trail and an expiry. A shared vault item in 1Password or Bitwarden is fine. Email and chat are not, because the secret then lives in two message histories neither of you controls.

If you want ongoing access without exposing SSH to the internet at all, an overlay network like Tailscale installed by the base role is a clean way to do it. The provisioning path still uses a public address for the first run, then the firewall closes it.

Troubleshooting a provisioning run

Ansible connects before cloud-init has finished

The most common failure by a wide margin. SSH comes up before cloud-init is done, Ansible connects, and then the first package task fails because cloud-init still holds the package manager lock. The error points at apt or dnf, which sends you looking in entirely the wrong place.

Fix it by blocking explicitly at the top of the first play, as in the verification snippet above. Check the state manually with:

cloud-init status --wait --long
sudo grep -i Traceback /var/log/cloud-init.log

One caveat that catches people: never put cloud-init status --wait inside a runcmd or bootcmd block. It will deadlock waiting for the process that is running it.

cloud-init appears to have done nothing

Two usual causes. Either the user-data was malformed, in which case cloud-init skipped it rather than erroring loudly, or the modules already ran on a previous boot and the semaphore files in /var/lib/cloud/instance/sem/ are preventing a rerun.

# what actually ran, and how long each stage took
cloud-init analyze show

# the human-readable output of scripts and package installs
sudo less /var/log/cloud-init-output.log

# on a throwaway test box only: wipe state and start over
sudo cloud-init clean --logs --reboot

Never run cloud-init clean on a production host. It removes cached state and the following boot will behave as a first boot.

Host key verification failed on a rebuilt server

Rebuild a machine on the same IP and every client with the old key in known_hosts refuses to connect. The tempting fix is to disable host key checking in ansible.cfg. Do not. That turns a real integrity check into a permanent no-op across every client you manage.

Remove the stale entry and pin the new one as part of the provisioning flow:

ssh-keygen -R 203.0.113.10
ssh-keyscan -H 203.0.113.10 >> ~/.ssh/known_hosts

Better still, if your provider exposes the host key fingerprint through its API or console output, compare it before you accept. That closes the trust-on-first-use gap properly.

The playbook reports changes on every single run

Almost always a shell or command task with no creates, changed_when or module equivalent. Ansible cannot know whether a raw command changed anything, so it assumes yes. Real changed counts are what let you spot drift, and a playbook that always reports twelve changes tells you nothing.

Run against an already-provisioned host in check mode. Anything that reports a change is either genuine drift or a task you need to fix:

ansible-playbook -i clients/acme/inventory/hosts.ini site.yml --check --diff

Common mistakes

• Putting the application build in runcmd. It runs once, never gets tested, and is invisible to every later change.
• One shared SSH key and one shared API token across all clients. Convenient right up to the moment it is not.
• Copying a role into a client directory to make one small change, instead of extracting a variable.
• Treating “the playbook exited zero” as proof the server is ready. It is proof no task threw.
• Local Terraform state on your laptop. It is a single point of failure that also cannot be handed over.
• Provisioning the server but not the DNS, TLS, monitoring and backups. A box nobody watches and nobody can restore is not finished.
• Automating the build and then fixing production by hand. One SSH session and you are back to a snowflake.

Best practices

• Rebuild before you need to. Destroy and rebuild a client’s staging host on a schedule. It is the only honest test of reproducibility.
• Version your roles and pin them per client. A shared role that everyone tracks on main means one commit changes six clients at once.
• Keep first boot minimal. If it can wait for Ansible, it should.
• Make monitoring part of provisioning. Whether that is Prometheus you run yourself, Grafana Cloud, or a lighter uptime service, the agent goes on in the base role, not as a follow-up ticket.
• Same for backups. An offsite target such as Backblaze B2 configured at build time, plus a restore test, or you have a backup you have never proven.
• Separate credentials per client, always. Keys, tokens, encryption keys, state backends.
• Write the runbook as you build. The provisioning repo’s README is the handover document. If it is accurate, the engagement can end cleanly.

Frequently asked questions

Do I need Terraform if I only manage a few VPS instances?

Not necessarily. If you are creating three servers a year through a provider’s control panel, the honest answer is that Terraform adds a state file to look after for very little return. Start with cloud-init and Ansible, which give you most of the reproducibility. Add Terraform or OpenTofu when the infrastructure around the server, such as firewalls, volumes, load balancers and DNS records, starts to be the part you keep getting wrong.

Should I build custom images instead of provisioning at boot?

Golden images cut boot time and remove dependency on package mirrors being up during a build. They also add a pipeline you have to maintain and rebuild for every security update. For client work at small scale, provisioning from a stock distribution image is usually the better trade, because the build stays readable and there is no stale image to forget about. Images earn their cost when you are launching machines often enough that boot time matters.

Terraform or OpenTofu for client projects?

Both work, and the configuration language is shared, so this is not a lock-in decision in the way it looks. Terraform has the deeper ecosystem and the commercial support path. OpenTofu is under the Linux Foundation with a licence that is unlikely to change, which is the stronger argument when the code will live in a client repository long after you have moved on. Ask the client whether they have a standard before you pick.

How do I automate server provisioning without exposing SSH to the internet?

The first run generally needs a reachable address, so scope the firewall rule to your own source IP for the duration of provisioning and close it in the same run. After that, reach the host through a bastion or an overlay network installed during the base role. Provider firewalls and host firewalls should both be in the automation, because relying on only one of them means a provider console change can silently open the box.

How do I stop configuration drift once the server is live?

Run the playbook in check mode on a schedule and look at what it reports. Any change on a host you have not touched is drift, and it is far cheaper to find it that way than during an incident. This only works if your tasks are properly idempotent, which is why the always-reports-changed problem above matters more than it looks.

What belongs in cloud-init versus Ansible?

cloud-init handles what must exist before you can connect: hostname, the login account, its SSH key, disabled password authentication, and a Python interpreter. Ansible handles everything you will ever want to change again. The dividing line is repeatability. cloud-init runs once and you cannot re-run it safely on a live host, so anything you might revisit belongs on the Ansible side.

How long should provisioning a new client server take?

Wall-clock time is the wrong measure. The useful measure is how much of it needs you. A build that takes twenty minutes but runs unattended from one command is in far better shape than a five-minute build with four manual steps, because the manual steps are where the divergence between clients gets introduced.

The one thing worth remembering

When you automate server provisioning for new clients, the deliverable is not the script. It is the ability to answer “can you rebuild this?” with yes, on any day, without opening a shell on the existing machine.

Everything else follows from that. Reproducibility gives you consistency across the fleet, because there is one definition. It gives you a clean handover, because the repository is the environment. It gives you the ability to take the next client on without dreading the build. And it costs the same afternoon that a single unplanned rebuild would have cost you anyway, just spent on your own schedule instead of at 2am.

Start with one client. Get their staging box rebuilding from scratch. Then make the second client’s build reuse the roles from the first. That is the whole path.

Need a provisioning setup you can actually hand over?

Most of the provisioning work I take on is not greenfield. It is a set of hand-built servers that work fine and nobody can reproduce. Things I can help with:

• Turning existing hand-built client servers into an Ansible role set, without a rebuild and without downtime
• Building a client onboarding repository: Terraform or OpenTofu, cloud-init, roles, per-client inventory and secrets
• Splitting shared credentials into per-client keys, tokens and state backends, with a revocation path that works
• Writing the verification playbook that decides whether a build is finished, and wiring it into CI
• Fixing idempotency so check-mode runs become a usable drift report instead of noise
• Preparing an environment for handover: runbook, access review, and a documented rebuild that the client can run themselves

If you want a second opinion on something specific, send the cloud-config, the playbook output, or a check-mode diff you cannot explain, and I will tell you what I see.

The post Automating Server Provisioning for New Clients Without Building a Snowflake Farm appeared first on John Nessime.

The reflex at that point is to reach for history rewriting. That is the wrong first move, and I will come back to why. But the bigger problem is the one that shows up six months after you clean up: teams migrate to Sealed Secrets or SOPS, tick the box, and quietly assume the whole class of problem is now solved. It is not. Encrypting a value before it enters Git closes exactly one door out of about five.

This post covers how to keep Kubernetes secrets out of Git in a way that survives contact with a real cluster: the three families of solution and where each one genuinely wins, the disaster-recovery trap that bites hardest, how to stop plaintext reaching the repo in the first place, and what to actually do about credentials already in your history.

The failure that actually bites: encrypted in Git, wide open in the cluster

Sealed Secrets and SOPS both terminate in the same place. The controller decrypts, the API server accepts, and what lands in etcd is an ordinary Kubernetes Secret. Ordinary means base64-encoded, which is an encoding, not a cipher. Anyone who can run kubectl get secret -o yaml in that namespace reads the value in one pipe through base64 -d.

So the honest description of what these tools buy you is narrower than the marketing suggests. They stop the credential appearing in a public object store with permanent history and a CDN in front of it. They do nothing about the cluster’s own blast radius.

Three things need to be true alongside whichever tool you pick:

• Encryption at rest is on. A stock Kubernetes control plane writes Secrets to etcd unencrypted unless you configure an EncryptionConfiguration, ideally with a KMS provider. Managed distributions vary. On EKS, envelope encryption is something you enable, not something you inherit. Check yours rather than assuming.
• RBAC on secrets is actually restrictive. A depressing number of clusters grant blanket get and list on secrets to service accounts that only ever needed one. That single verb turns a namespace compromise into a credential dump.
• Secrets are mounted as files, not shoved into environment variables where you can help it. Environment variables leak through /proc/<pid>/environ, through anything that dumps the environment on crash, and through every debug endpoint an application framework has ever shipped. A projected volume with tight permissions is the quieter option.

Quick sanity check on the RBAC side, which takes about ten seconds and surprises people:

# Can the default service account in a namespace read every secret in it?
kubectl auth can-i get secrets 
  --namespace production 
  --as system:serviceaccount:production:default

If that returns yes, your Git hygiene is not the weakest link in the chain.

Already leaked? Rotate first, rewrite second, and understand what rewriting does not do

Order matters here more than technique. GitHub’s own guidance is blunt about it: once a credential has been pushed, revoke or rotate it first, because that step alone removes the attacker’s ability to use it. History surgery is the optional follow-up, not the fix.

The reason is that a force-push does not reach everywhere the commit went. Objects survive in forks, which inherit the full object graph at fork time. They survive in colleagues’ clones. On GitHub they remain reachable by SHA through cached views and through pull requests that referenced them, and purging those caches is a support request, not a git command. If the repo was public even briefly, treat the credential as burned and move on.

When you do rewrite, use git filter-repo. It has replaced filter-branch as the recommended tool and it is dramatically faster on any repo with real history.

# Work on a bare mirror, never your day-to-day clone.
git clone --mirror git@github.com:org/infra.git infra-clean.git
cd infra-clean.git

# Strip one path from every commit that ever touched it.
git filter-repo --path clusters/production/values.yaml --invert-paths

# filter-repo deletes the origin remote on purpose, so you cannot
# force-push a half-finished rewrite by muscle memory. Add it back.
git remote add origin git@github.com:org/infra.git
git push --force --all
git push --force --tags

Then the unglamorous part: close or merge open pull requests before you start, because every SHA downstream of the rewrite changes. Tell every collaborator to delete their clone and clone again. A git pull will happily drag the old objects back in.

Two families of solution, and how to tell which one you are in

Every approach to this problem is either encrypt the value and commit the ciphertext, or commit a pointer and fetch the value at runtime. Everything else is implementation detail. The choice is mostly about whether you already run a secret store, and whether you can tolerate the CLI step in the middle of your workflow.

Sealed Secrets: the lowest-friction way to start

A controller in the cluster holds an RSA private key and publishes the matching certificate. The kubeseal CLI encrypts a Secret manifest against that certificate and produces a SealedSecret custom resource. Only that controller can decrypt it, so the resulting YAML is safe to commit.

# Fetch the controller's public certificate. This is safe to commit;
# it can only encrypt, never decrypt.
kubeseal --fetch-cert 
  --controller-name=sealed-secrets 
  --controller-namespace=kube-system 
  > pub-cert.pem

# Build a Secret manifest locally. --dry-run=client means it is rendered
# and never sent to the API server.
kubectl create secret generic db-auth 
  --from-literal=password='<value>' 
  --dry-run=client -o yaml > db-auth.yaml

# Encrypt against the cert. Offline, so no cluster access needed here.
kubeseal --cert pub-cert.pem --format yaml < db-auth.yaml > db-auth-sealed.yaml

rm db-auth.yaml

Because you can hand out pub-cert.pem, developers can seal secrets without any cluster credentials at all. That is the underrated part, and it is the reason this is the one I reach for first on a small team with no existing vault.

Two details that catch people. First, sealing is scoped by default: a SealedSecret is bound to its exact name and namespace, and moving the file to a different namespace makes it undecryptable. That is deliberate, it stops a developer sealing a secret into a namespace they should not have access to, and you can relax it with --scope namespace-wide or --scope cluster-wide if you genuinely need to. Second, the Helm chart names the controller sealed-secrets while the CLI looks for sealed-secrets-controller by default, so --controller-name is not optional in practice.

Where it stops being the right answer: multi-cluster. Each cluster generates its own key pair, so a sealed file is not portable, and a fleet means either copying private keys around or maintaining per-cluster ciphertext for the same value. Rotation is also manual. There is no dynamic secret story here at all.

SOPS with age: encrypt files, not just Kubernetes objects

SOPS encrypts the values inside a structured file and leaves the keys readable, so a diff still tells you which field changed even if it cannot tell you what it changed to. Paired with age, which uses short X25519 keys instead of the GPG keyring experience, it is pleasant to live with.

A .sops.yaml at the repo root drives everything through path matching, which is how you get per-environment key separation without anyone having to remember flags:

creation_rules:
  - path_regex: clusters/production/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # production recipient
  - path_regex: clusters/staging/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # staging recipient

The encrypted_regex line is the one worth understanding rather than copying. Without it SOPS encrypts everything including metadata, and you lose the readable-diff property that made this approach attractive. Restricting it to data and stringData keeps names, namespaces and labels reviewable in a pull request.

Flux decrypts SOPS natively: you point a Kustomization at a Secret holding the age private key via decryption.provider: sops and it handles the rest. Argo CD does not have equivalent built-in support, so you bolt it on with a config management plugin such as KSOPS or the Argo CD Vault Plugin. Worth confirming against current Argo CD docs before you commit to it, because plugin mechanics there have changed more than once.

SOPS also handles files that are not Kubernetes manifests: Terraform variable files, Ansible vars, plain .env. If you have config living outside the cluster, that reach is a real advantage over Sealed Secrets.

External Secrets Operator: stop putting the value in Git at all

The third approach sidesteps encryption entirely. You commit a reference, and a controller resolves it against a real secret store: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or one of the newer hosted options like Doppler, Infisical or 1Password. The repo contains a path, never a value.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-auth
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-auth
  data:
    - secretKey: password
      remoteRef:
        key: production/postgres
        property: password

This is the only one of the three that gives you rotation without a commit. Change the value in Vault, wait for refreshInterval, and the cluster Secret updates. It also gives you a real audit trail, because reads happen against a system built to log them.

The cost is honest and worth stating. You have introduced a runtime dependency: if the store is unreachable, new workloads cannot get their credentials. You still have a bootstrap credential problem, because the operator needs to authenticate to the store somehow, and that root of trust has to get into the cluster by some other path. And Git is no longer the complete description of your system, which is a genuine philosophical cost if GitOps purity matters to you.

One live upgrade trap. The external-secrets.io/v1beta1 API was removed in ESO 0.17.0. Before that, in the 0.16 series, a conversion webhook served both versions and rewrote stored objects to v1, which produced permanent drift in Argo CD for anyone whose Git manifests still said v1beta1. Argo sees v1 in the cluster, Git says v1beta1, and it reconciles forever. Update your manifests to v1 before you cross that boundary, not after.

Secrets Store CSI Driver: when a Secret object is one object too many

Worth knowing about even if you do not use it. The CSI driver mounts values from an external store directly into the pod as a volume, so no Kubernetes Secret object needs to exist at all. That closes the etcd exposure completely. It costs you the ability to consume the value as an environment variable, and it means the secret is only available to pods that mount it, which is either exactly what you want or a nuisance depending on the workload. Use it where the compliance posture demands that etcd never sees the value.

The key you forgot to back up

This is the failure I would put money on, and it is the one nobody rehearses. Every encrypt-into-Git approach concentrates all your risk into one small piece of key material. Lose it and your repository turns into a directory of well-organised noise.

Concretely: rebuild a cluster from scratch, install a fresh Sealed Secrets controller, and it generates a new key pair. Every SealedSecret in your repo is now undecryptable. Your GitOps repo, the one that was supposed to let you recreate everything, cannot recreate anything that needed a credential. You find this out during the incident, not before it.

• Back up the sealing key out of band. The controller’s private key lives in a Secret in its namespace. Export it, encrypt it, and store it somewhere that is not the cluster and not the repo. A password manager or an offline copy is fine. Test the restore.
• Same for age. The private key is one line of text. Its size makes it feel unimportant. It is the entire thing.
• Add a second recipient for anything that matters. SOPS encrypts the data key once per recipient, so listing both an age key and a KMS key means either path can decrypt. The developer works with age locally, the cluster decrypts through KMS, and no single lost key is fatal.
• Know your re-key procedure before you need it. For SOPS that is sops updatekeys across every file matched by the rule. For Sealed Secrets it means re-sealing everything against the new certificate. Neither is hard. Both are miserable to work out under pressure.

Stopping plaintext from reaching the repo in the first place

All of the above assumes the encryption step happens. The commit that leaks is always the one where somebody skipped it, usually while debugging at the end of a long day.

1. Put a scanner in the pre-commit hook. gitleaks and trufflehog both work well here. Local hooks are bypassable, which is fine, because their job is catching accidents rather than stopping a determined person.
2. Run the same scan in CI, on the whole history, not just the diff. The diff-only scan misses everything that predates the day you added the scanner.
3. Turn on server-side push protection. GitHub’s push protection rejects known credential patterns at push time. GitLab and platforms like GitGuardian offer equivalents. This is the layer that actually holds, because it does not depend on anyone’s local setup.
4. Make the plaintext path short-lived by construction. Never write a plaintext manifest to a tracked path. Use /tmp, or pipe straight into the encrypt step so the value never lands on disk at all.
5. Add *.dec.yaml, secret.yaml and friends to .gitignore as a convention. Not a control, just one less way to be careless.
6. Remember your shell history. A --from-literal with a real password ends up in ~/.bash_history in cleartext. Use --from-file, or read from a variable, or prefix the command with a space if your shell is configured to skip those.

Troubleshooting the four things that go wrong

The SealedSecret applies cleanly but no Secret appears

Almost always a scope mismatch. The resource was sealed for one name or namespace and applied to another. The API server accepts the custom resource happily; the controller is where the failure surfaces.

# The controller logs are the only place this error lives.
kubectl logs -n kube-system deploy/sealed-secrets --tail=50

# Confirm what the CRD actually thinks its name and namespace are.
kubectl get sealedsecret db-auth -n production -o yaml

The second common cause: the file was sealed against a certificate from a different cluster, or from before a key rotation. Re-fetch the cert and re-seal.

The ExternalSecret sits there doing nothing

The status conditions carry the real message, and the events under describe usually name the provider error directly.

kubectl get externalsecret db-auth -n production
kubectl describe externalsecret db-auth -n production
kubectl describe clustersecretstore vault-backend

Check the store before the secret. If the SecretStore is not Ready, nothing referencing it will ever sync, and the error you want is on the store, not the ExternalSecret. When using a ClusterSecretStore, remember that any secret reference inside it needs an explicit namespace, because a cluster-scoped object has no namespace of its own to fall back on.

Argo CD or Flux reports permanent drift

Two usual causes. Either an API version conversion is rewriting stored objects to a version your Git manifests do not use, as with the ESO v1beta1 removal above, or a controller is adding fields to the generated Secret that your reconciler then tries to remove. Compare the live object against the rendered manifest field by field before you start changing anything.

SOPS decryption fails only in the cluster

You encrypted for one recipient and the cluster holds a different key. This is the classic outcome of listing multiple recipients in .sops.yaml but running sops --encrypt with an explicit --age flag that overrides the rule. Decrypt locally to prove your key works, then check which recipients are actually listed in the file’s SOPS metadata block. The recipient list is stored in the file, in the clear, which makes this a fast thing to diagnose.

Common mistakes

• Rewriting history before rotating the credential, and treating the rewrite as the remediation.
• Assuming a private repo is safe enough. Private repos get forked internally, cloned onto laptops, and made public by accident.
• Committing the encrypted file and the plaintext source next to it, because the encrypt step wrote to a new filename and nobody deleted the original.
• Never backing up the sealing key or the age identity.
• Leaving encryption at rest off, so the credential moves from a Git leak to an etcd snapshot leak.
• Encrypting the entire manifest with SOPS instead of just data and stringData, and losing every useful diff.
• Using one age key or one sealing controller for all environments, so a staging compromise reaches production ciphertext.
• Piping a Helm release’s rendered output into a debug artefact in CI. Helm renders secrets in the clear, and CI artefacts are frequently world-readable inside an organisation.

How I would actually decide

Skip the feature matrix and answer three questions in order.

1. Do you already run a secret store? If Vault or a cloud secret manager is already in your environment and someone owns it, use External Secrets Operator. You get rotation and audit for almost no additional conceptual load, and the argument is over.
2. Do you have config outside Kubernetes? Terraform variables, Ansible vars, application .env files. If yes, SOPS with age, because one tool covers all of it and the per-path rules give you environment separation for free.
3. Neither? Sealed Secrets. One cluster, a handful of secrets, a team that wants this solved this afternoon. It is the shortest distance to a repo with no plaintext in it, and migrating away later is straightforward because the plaintext values are recoverable from a running cluster.

Whichever you land on, the cluster-side work is the same and it is not optional: encryption at rest, tight RBAC on secrets, files rather than environment variables, and a tested backup of whatever key material the scheme depends on. On a managed control plane from a provider like DigitalOcean, Linode or a hyperscaler, the encryption-at-rest configuration is partly theirs and partly yours, so read the specific documentation rather than trusting a general answer. On a self-managed cluster running on your own VPS instances, for instance a k3s setup on InterServer or similar, all of it is yours and none of it is on by default.

FAQ

Is base64 in a Kubernetes Secret a form of encryption?

No. Base64 is an encoding that exists so binary values survive YAML parsing. It is reversed with a single command and provides no confidentiality whatsoever. Anyone reading the manifest reads the secret.

Should I use Sealed Secrets or External Secrets Operator?

If you already run Vault, AWS Secrets Manager or an equivalent, use External Secrets Operator: you get rotation without commits and a proper audit trail. If you have no secret store and no appetite to run one, Sealed Secrets is a faster path with fewer moving parts. The trade-off is that Sealed Secrets has no rotation story and its keys are per-cluster.

What happens if I lose the Sealed Secrets private key?

Every SealedSecret in your repository becomes permanently undecryptable, and you regenerate every credential from source. This is the single most common way this setup fails in practice. Back the key up somewhere outside the cluster and the repo, and test restoring it.

Do I still need to rewrite Git history after rotating a leaked credential?

Often not. Once the credential is revoked it cannot be used, which usually resolves the actual risk. Rewriting is worth the disruption when the value is not rotatable, when it is personal data rather than a credential, or when a compliance process requires it. Weigh it against breaking every open pull request and forcing everyone to re-clone.

Can Argo CD decrypt SOPS files on its own?

Not natively, unlike Flux, which has SOPS decryption built into its Kustomize controller. With Argo CD you add a config management plugin such as KSOPS or the Argo CD Vault Plugin. Check the current Argo CD documentation for the supported plugin mechanism before building around it.

Is it safe to commit the Sealed Secrets certificate or the age public key?

Yes, and committing them is the point. Both are public halves of an asymmetric pair and can only encrypt. Putting them in the repo is what lets developers seal new secrets without cluster access. The private halves never go anywhere near Git.

The one thing worth remembering

If you take one idea away, make it this: the work to keep Kubernetes secrets out of Git and the work to keep them safe in the cluster are two separate jobs, and finishing the first one feels a lot like finishing both. It is not. A repo full of SealedSecret resources sitting in front of an unencrypted etcd with permissive RBAC has moved the exposure, not removed it.

Pick the approach that matches what you already run. Back up the key on day one, not after the first rebuild. Turn on encryption at rest. And when something does leak, rotate before you reach for filter-repo, because rotation is the part that actually stops the bleeding.

Need help sorting out secrets in your cluster?

This is work I do regularly, and most of it is less dramatic than it sounds once someone has done it before. Things I can help with:

• Auditing a GitOps repo for committed credentials, including full history, and producing a prioritised rotation list rather than a wall of scanner output
• Setting up Sealed Secrets, SOPS with age, or External Secrets Operator against Vault or a cloud secret manager, including the key backup and restore procedure
• Migrating an existing repo off plaintext or off a scheme that no longer fits, without a big-bang cutover
• Fixing Argo CD or Flux reconciliation loops caused by secret controllers and API version conversions
• Enabling and verifying etcd encryption at rest, and tightening RBAC on secrets across namespaces
• Adding pre-commit and CI secret scanning that people will not immediately disable because of false positives

If you have a controller log, a stuck ExternalSecret, or a scanner report you are not sure how to triage, send it over and I will tell you what I make of it.

The post Keep Kubernetes Secrets Out of Git Without Kidding Yourself appeared first on John Nessime.

None of that is a bug. AWS optimises the empty account for getting something running in ten minutes. The problem is that most accounts never get corrected afterwards, and eighteen months later there is production traffic sitting on top of decisions nobody made deliberately.

An AWS account baseline is the set of things you lay down before the first workload, so the account starts from a defensible posture instead of a blank console. This walks through the decisions that actually matter, why each one is the way it is, and what it costs. I have put a working reference implementation on GitHub — terraform-aws-baseline — so you can read the real Terraform rather than take my word for any of it.

What a baseline is, and what it is not

A baseline is not a landing zone in the Control Tower sense, and it is not a compliance framework. It does not make you SOC 2 compliant and nothing that fits in one repository will.

What it is: the boring foundational layer that everything else sits on. State storage, network segmentation, identity, and the guardrails that stop a careless afternoon becoming an incident. If you get these right once, every workload that lands afterwards inherits them for free.

The structure that works is three layers. A bootstrap stack that creates the state backend and runs exactly once. Reusable modules for the things every environment needs. Thin per-environment roots that compose those modules with different parameters — dev gets one NAT gateway and seven-day log retention, prod gets one per availability zone and a year.

Decision one: no long-lived AWS credentials, anywhere

Static access keys in CI are the single most common way AWS accounts get compromised. They live in a secrets store, they get copied into a second one, somebody echoes one into a build log, and nobody rotates them because rotation breaks things.

GitHub Actions can assume an AWS role using a short-lived OIDC token that GitHub signs and AWS verifies. There is no key to store, nothing to leak, and nothing to rotate.

The important part is the trust policy, and one word in it decides whether this is a security improvement or a vulnerability:

data "aws_iam_policy_document" "ci_assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    # StringEquals, not StringLike. This is the whole ballgame.
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values = [
        "repo:${var.github_org}/${var.github_repo}:ref:refs/heads/${var.github_branch}"
      ]
    }
  }
}

Swap StringEquals for StringLike and add an asterisk, and you have widened that trust from one branch of one repository to whatever the wildcard covers. Get sloppy enough — a bare repo:* — and any repository on GitHub can assume your role. This is not theoretical; it is a recurring finding in real audits.

Scope it to one repository and one branch. If you need more, add more statements, explicitly.

Decision two: treat Terraform state as a secret, because it is

Terraform state contains every attribute of every resource it manages. Database passwords, generated keys, connection strings. Marking a variable sensitive redacts it from console output and changes nothing about what is written to the state file.

So the state bucket gets four things: server-side encryption with a customer-managed KMS key that rotates, versioning so a corrupted state can be recovered, all four public access block flags, and a bucket policy that refuses plaintext connections outright.

resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

data "aws_iam_policy_document" "deny_insecure_transport" {
  statement {
    sid       = "DenyInsecureTransport"
    effect    = "Deny"
    actions   = ["s3:*"]
    resources = [
      aws_s3_bucket.state.arn,
      "${aws_s3_bucket.state.arn}/*",
    ]

    principals {
      type        = "*"
      identifiers = ["*"]
    }

    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }
}

All four public access block flags, not two. They cover different paths — ACLs and bucket policies, existing objects and future ones — and setting a subset leaves a gap that reads as protected on a dashboard.

There is a chicken-and-egg problem here that catches everyone: the stack that creates the state bucket cannot store its own state in a bucket that does not exist yet. You run it once with local state, then migrate:

cd bootstrap
terraform init
terraform apply
terraform output          # bucket, lock table, KMS key, region

# Now move bootstrap's own state into the bucket it just created
terraform init -migrate-state

Decision three: shut the default security group

Every VPC gets a default security group that permits all traffic between members and all outbound traffic. Anything launched without an explicit security group lands on it. That is a quiet failure mode — nothing errors, the instance works, and it has more network reach than anyone intended.

Terraform can adopt and empty it:

resource "aws_default_security_group" "default" {
  vpc_id = aws_vpc.this.id

  # No ingress and no egress blocks. Terraform strips AWS's rules,
  # so anything that accidentally lands here can reach nothing.
}

Four lines, and misconfiguration becomes loud instead of silent.

Decision four: three subnet tiers, and the bottom one has no way out

The standard two-tier split — public and private — leaves databases in the same tier as application servers. Both can reach the internet through NAT. That is fine right up until something in the data tier is compromised and quietly exfiltrates over 443, which looks like ordinary egress traffic.

• Public — NAT gateways and load balancers. Route to the internet gateway.
• Private — workloads. Outbound through NAT, no inbound from the internet.
• Database — data stores. No default route at all.

The third tier is enforced by absence rather than by a rule:

resource "aws_route_table" "database" {
  vpc_id = aws_vpc.this.id

  # Deliberately no 0.0.0.0/0 route. Not to the IGW, not to NAT.
  # Nothing here can reach the internet, and nothing can reach it.
}

No security group to misconfigure, no NACL to get wrong. The route simply does not exist. That is a much stronger guarantee than a rule someone can widen during an incident and forget to narrow afterwards.

Decision five: gateway endpoints, the one genuinely free win

S3 and DynamoDB gateway endpoints cost nothing. Not “cheap” — zero. They add a route so traffic to those services stays on the AWS backbone rather than going out through NAT.

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.this.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"

  route_table_ids = concat(
    aws_route_table.private[*].id,
    aws_route_table.database[*].id,
  )
}

Two benefits, and the second is the one people miss. Traffic to S3 never traverses NAT, so you stop paying per-gigabyte processing on it. And the isolated database tier can still reach S3 and DynamoDB despite having no internet route, because the endpoint is a route within the VPC rather than a path out of it.

Your Terraform state lives in S3. Without this endpoint, every plan and apply pushes state through the NAT gateway and you are billed for the privilege.

Decision six: SSM Session Manager, not SSH

Port 22 open anywhere means key pairs to distribute, a bastion host to patch, and an audit trail that is whatever auth.log happens to contain.

resource "aws_iam_role_policy_attachment" "ssm_core" {
  role       = aws_iam_role.instance.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

That one attachment gets you shell access with no inbound ports, no key pairs, no bastion. Access is IAM-authorised, so revoking someone is a permissions change rather than a key rotation, and every session can be logged centrally.

Worth knowing the caveat: SSM agents reach the SSM endpoints over the internet by default, so instances in the private tier still need NAT for this. Removing NAT entirely means adding interface endpoints for ssm, ssmmessages and ec2messages, and those cost roughly seven dollars a month per availability zone each.

Decision seven: pin actions to commit SHAs

A Git tag is a movable pointer. If an action’s repository is compromised, an attacker can repoint v4 at whatever they like, and every workflow referencing that tag runs it on the next build — with your OIDC role attached.

steps:
  # A tag can be moved. A commit SHA cannot.
  # Dependabot updates these pins and shows you the diff.
  - uses: actions/checkout@<full-40-character-sha>   # v4.2.2

The comment matters as much as the pin. Without it nobody can tell at a glance which version they are looking at, and the pin becomes something people update blindly.

What it actually costs

Almost all of it is NAT gateways. Everything else is rounding.

• NAT gateways — roughly $32 a month each, plus about $0.045 per gigabyte processed. One in dev, one per availability zone in prod.
• VPC, subnets, route tables, internet gateway — free.
• S3 and DynamoDB gateway endpoints — free.
• KMS keys — about a dollar a month each.
• Flow logs — around $0.50 per gigabyte ingested. Retention is the lever: seven days in dev, a year in prod.
• State bucket and lock table — cents.

Ballpark, that lands around $35 a month for dev and $105 for prod, dominated entirely by NAT. If you only need the isolated tiers and nothing requires outbound internet, dropping NAT takes dev close to free.

Per-AZ NAT in production is an availability decision, not a security one. A single NAT gateway is a single point of failure for outbound traffic across every AZ that routes through it.

Gates that run before any credentials exist

Most of what goes wrong in Terraform is catchable statically. Formatting, syntax, provider misuse, and known-insecure patterns can all be found without touching an AWS account, which means they can run on every commit and in CI without the pipeline holding any cloud credentials at all.

# On every commit, locally
pre-commit install        # gitleaks, fmt, validate, tflint, checkov, terraform-docs

# The same gates CI runs
make all                  # fmt + validate + tflint + checkov

Running the identical command locally and in CI is the part that matters. When those two drift, people stop trusting the local check and start pushing to see what the pipeline says, and the fast feedback loop you built quietly stops existing.

Note also what is missing: there is no terraform plan in that pipeline. Plan needs real credentials, and a static-only pipeline is one that cannot leak anything. A production setup would add a plan job running under the OIDC role and posting to the pull request — but that is a deliberate step up in blast radius, not a default.

What a baseline like this deliberately leaves out

Being clear about scope is more useful than pretending completeness. The reference implementation covers preventative controls and stops there. Missing, on purpose:

• Detective controls. No CloudTrail, Config, GuardDuty or Security Hub. A complete account baseline includes them; this is the preventative half.
• Compute. There is an instance role and profile, but nothing uses them. No ASG, no ECS or EKS, no load balancer.
• Interface endpoints. Only the free gateway ones. Fully private SSM needs paid interface endpoints.
• Multi-region and DR. Versioning covers state recovery. There is no cross-region replication and no failover story.
• Policy-as-code tests. Checkov catches known misconfigurations; there are no Terratest or Conftest assertions on module contracts.

A baseline you can describe the edges of is more useful than one advertised as complete. You know exactly what you still have to build.

Common mistakes

• Using StringLike with a wildcard in an OIDC trust policy. Scope to one repository and one branch.
• Assuming sensitive = true keeps values out of state. It does not.
• Setting two of the four S3 public access block flags and considering the bucket locked down.
• Leaving the default security group as AWS shipped it.
• Putting databases in the same tier as application servers because two tiers felt like enough.
• Skipping gateway endpoints and paying NAT processing charges on your own state file.
• Referencing actions by tag and assuming a tag is immutable.
• Committing backend.hcl or terraform.tfvars. State bucket names embed the account ID.

Frequently asked questions

Is an AWS account baseline the same as a landing zone?

No. A landing zone in the Control Tower sense governs many accounts — organisational units, service control policies, account vending. A baseline is what you lay down inside a single account. Baselines are what you need when you have one or two accounts and Control Tower would be enormous overkill.

Why not just use Control Tower?

Control Tower is a good answer for a multi-account organisation with governance requirements. It is a lot of machinery for a team running a handful of accounts, and it hides the decisions rather than making them legible. Writing the baseline yourself means you can read every choice and change any of them.

Do I really need a separate database subnet tier?

It costs nothing and closes an entire exfiltration path. If your data tier has no route to the internet, a compromised database cannot phone home regardless of what the attacker does with it. Two tiers is not wrong, but the third is close to free.

How do I cut the NAT gateway cost?

Use one NAT for non-production instead of one per AZ. Add gateway endpoints so S3 and DynamoDB traffic bypasses NAT entirely. Beyond that, work out what genuinely needs outbound internet — often less than you assume — and consider dropping NAT for environments that do not.

Can I use this with OpenTofu?

Yes. It is standard HCL against the AWS provider, so it runs on either. Swap terraform for tofu in the commands and everything else is the same.

Where does the OIDC provider itself live?

It is account-global, so only one environment can create it and the others reference it. That is workable but awkward. In a real organisation it belongs in a separate account-level stack that runs before any environment.

Wrapping up

None of the decisions in an AWS account baseline are individually clever. Scope the trust policy. Encrypt the state. Empty the default security group. Give the data tier no way out. Take the free endpoints. Use SSM instead of SSH. Pin your actions. Each one is a few lines of Terraform.

What makes them valuable is doing them before the first workload arrives, because retrofitting network segmentation into a running account is a project, and doing it at the start is an afternoon.

The full implementation is on GitHub at JohnNessime/terraform-aws-baseline, MIT licensed. It is a reference implementation rather than something that has served production traffic — a worked example of how I would lay an account down, with the reasoning written out in the docs. Read it, disagree with parts of it, take what is useful.

Need a baseline laid down properly?

I work with teams on AWS foundations — account structure, network segmentation, IAM that does not rely on long-lived keys, Terraform layouts that survive more than one engineer, and CI pipelines that catch problems before they reach an account.

If you would rather have this done once and done right, you can hire me on Upwork.

The post A Secure-by-Default AWS Account Baseline in Terraform appeared first on John Nessime.

The Pulumi vs Terraform question got harder in the last two years, not easier. The licence changed, the community forked, IBM bought HashiCorp, HashiCorp killed its own general-purpose-language option, and Pulumi started speaking HCL. A lot of the comparison articles you will find were written before any of that happened.

This is an attempt at an honest read of where things actually stand, what the real trade-off is once you strip away the marketing, and how to pick a lane you will not regret in eighteen months.

What actually changed

A short timeline, because the current landscape only makes sense against it.

• August 2023 — HashiCorp relicensed Terraform from MPL 2.0 to the Business Source License. Source-available, free for provisioning your own infrastructure, but you cannot build a competing commercial product on it.
• January 2024 — OpenTofu shipped 1.6.0, a fork of the last MPL-licensed code, governed by the Linux Foundation.
• February 2025 — IBM completed its acquisition of HashiCorp, valued at around $6.4 billion.
• December 2025 — HashiCorp deprecated CDK for Terraform and archived the repository. The official way to write Terraform in a general-purpose language no longer exists.
• Also December 2025 — Pulumi announced native HCL support and the ability to work with Terraform and OpenTofu state.

Those last two matter more than the licence drama, and they point in opposite directions. HashiCorp closed the door on writing Terraform in a real language. Pulumi opened the door to writing Pulumi in HCL. Whatever else you conclude, “HCL versus real languages” is no longer a clean line between two products.

The real axis: a configuration language or a programming language

Strip out the licensing and the corporate ownership and one genuine difference remains. HCL is a purpose-built configuration language that describes desired state. Pulumi runs an actual program that constructs a resource graph, and then applies it.

Here is the same trivial thing in both. Terraform:

resource "aws_s3_bucket" "logs" {
  for_each = toset(var.environments)
  bucket   = "acme-logs-${each.key}"

  tags = {
    Environment = each.key
    ManagedBy   = "terraform"
  }
}

And Pulumi, in TypeScript:

import * as aws from "@pulumi/aws";

const environments = ["dev", "staging", "prod"];

const buckets = environments.map(env =>
  new aws.s3.Bucket(`logs-${env}`, {
    bucket: `acme-logs-${env}`,
    tags: {
      Environment: env,
      ManagedBy: "pulumi",
    },
  })
);

For something this small the difference is aesthetic. It stops being aesthetic around the third nested conditional.

What you actually gain with a general-purpose language

• Real abstractions. Classes, functions, inheritance. A Pulumi component that encapsulates “our standard service” is just a class, and it composes the way code composes.
• Real testing. Jest, pytest, Go’s testing package. You can unit test infrastructure logic with the same tools and the same CI as the rest of your codebase.
• The package ecosystem. Need to parse a CSV of subnets, call an internal API, or do something genuinely awkward? It is an npm install rather than an external data source and a shell script.
• IDE support that works. Type checking, refactoring, jump-to-definition. HCL tooling has improved a lot but it is not the same thing.

What you lose, which comparison posts skip

This is the part worth reading twice, because it is where the regret comes from.

• Constraint is a feature. HCL is limited on purpose. When a language cannot express clever things, nobody writes clever things. Give a team a real language and someone will eventually build a metaprogramming layer that generates resource names from a database lookup, and reviewing that pull request will be nobody’s idea of a good afternoon.
• Readability across skill levels. A network engineer who has never written TypeScript can usually read HCL and tell you what it provisions. The reverse is not true. If your infrastructure is touched by people who are not primarily programmers, that matters enormously.
• The hiring pool. There are far more engineers with Terraform on their CV than Pulumi. That is not an argument about quality, it is an argument about how long it takes to backfill a role.
• A dependency on a runtime. Pulumi programs need Node, Python, or Go present and correct in every environment that runs them, including CI. It is one more thing that can drift.
• Asynchrony leaks into the model. Resource outputs are not plain values. They are futures, and you have to handle them as such. Every Pulumi team hits this on day two.

# This does not do what a newcomer expects. bucket.id is an Output,
# not a string, so you get an object reference in the name.
# bucket_name = f"logs-for-{bucket.id}"

# Correct: work inside apply, where the value is resolved.
bucket_name = bucket.id.apply(lambda bid: f"logs-for-{bid}")

None of these are dealbreakers. They are just the bill that arrives later, and it is worth reading before you sign.

State, providers, and the things that actually break

State management

Both tools keep a state file mapping your code to real resources, and both will ruin your afternoon if that file gets corrupted or out of sync.

Terraform and OpenTofu use pluggable backends: S3 with DynamoDB locking, Azure Blob, GCS, Consul, or HCP Terraform. OpenTofu added client-side state encryption, which Terraform does not have natively and which is a genuine advantage if your compliance people care about what is sitting in that bucket.

Pulumi defaults to Pulumi Cloud, which is free for individuals and paid for teams. It also supports self-managed backends, and you should know this because the default sometimes gets mistaken for a requirement:

# Self-managed state in S3, no Pulumi Cloud account involved
pulumi login s3://my-pulumi-state-bucket

# Or entirely local, useful for experiments
pulumi login --local

Pulumi encrypts secrets in state by default. With Terraform, values marked sensitive are redacted in output but still sit in plaintext in the state file, which surprises people who assumed otherwise.

The provider ecosystem, and a myth worth killing

You will read that Pulumi has a much smaller provider ecosystem. That was true once and is mostly no longer the point, because Pulumi bridges Terraform providers. If a Terraform provider exists, you can almost certainly use it from Pulumi, either pre-built in the Pulumi Registry or generated on demand.

Two caveats that are real:

• Bridged providers can lag upstream releases. If you need a resource type that shipped last week, check before committing.
• When something breaks deep in a bridged provider, you are debugging across a translation layer. That is a worse afternoon than debugging the provider directly.

Pulumi also ships native providers for Kubernetes and Azure, generated from the platform APIs, which give same-day coverage of new resources rather than waiting on a bridge.

Policy and governance

Terraform’s first-class answer is Sentinel, which is tied to HCP Terraform. OpenTofu users generally reach for OPA or Conftest. Pulumi has CrossGuard, where policies are written in the same languages as everything else.

If you are in a regulated environment and your auditors already know what Sentinel is, that is a heavier thumb on the scale than any language preference.

Licensing, since it is what forced the question

• Terraform — BSL 1.1. Source-available. Fine for provisioning your own infrastructure. Not fine for building a product that competes with HashiCorp. Each release converts to an open licence four years after publication, but only that release.
• OpenTofu — MPL 2.0, OSI-approved, Linux Foundation governance. No single vendor can relicense it.
• Pulumi — Apache 2.0 for the engine and CLI. Permissive, with a patent grant. Pulumi Corp monetises through Pulumi Cloud rather than through the licence.

For the overwhelming majority of teams, the BSL changes nothing about daily work. You are provisioning your own infrastructure and that is explicitly permitted. The concern is not today’s terms, it is that a single vendor demonstrated it can change them, and that vendor is now part of IBM.

If your objection to Terraform is licensing rather than language, the answer is OpenTofu, not Pulumi. Switching paradigms to solve a legal problem is an expensive way to do it.

Choosing a lane

Three coherent positions. Pick the one that matches your constraints rather than the one that sounds most modern.

Lane one: OpenTofu

The default for most teams already running Terraform who are uneasy about the licence. Migration is genuinely close to free: same HCL, same providers, same state format, same commands.

# Back up state first. Always.
terraform state pull > backup-$(date +%F).tfstate

# Then initialise with tofu against the same backend
tofu init
tofu plan

A clean plan with no proposed changes means you are done. That is the whole migration for most codebases.

Choose this if: you have existing Terraform, HCL is working for you, and you want open governance without retraining anyone.

Lane two: Terraform with HCP

Still the biggest ecosystem, the most documentation, the most Stack Overflow answers, the most people who already know it. Stacks and ephemeral resources are Terraform-only. Sentinel is Terraform-only. If you are deep in HCP Terraform and it works, the licence is an abstract concern rather than a practical one.

Choose this if: you are in a regulated industry, you need vendor support with a contract behind it, or your organisation is standardising on IBM and Red Hat tooling anyway.

Lane three: Pulumi

The strongest case for Pulumi is not “real languages are nicer”. It is when your infrastructure genuinely needs logic that HCL fights you on: dynamic topologies, infrastructure derived from application code, platform teams building abstractions for other teams to consume. If you are writing a self-service platform rather than a set of environments, a real language stops being a preference and starts being the right tool.

Choose this if: your team is primarily software engineers, your infrastructure has real branching logic, or you are building a platform product rather than provisioning a fixed estate.

Do not choose it because writing loops in HCL annoyed you once.

What migrating actually looks like

Terraform to OpenTofu is a swap. Terraform to Pulumi is a refactor, and anyone telling you otherwise is selling something.

Pulumi does provide real tooling for it:

# Convert HCL into a Pulumi program in your chosen language
pulumi convert --from terraform --language typescript

# Adopt the resources described by an existing state file
pulumi import --from terraform ./terraform.tfstate

Pulumi can also consume Terraform modules directly, so existing module investment is not automatically wasted. That is the single most useful fact for anyone considering a gradual move.

A sane sequence:

1. Pick one small, low-risk stack. Not the production VPC.
2. Convert it, import the state, and run a preview. The only acceptable result is no changes.
3. Refactor the generated code. Converted output is mechanically correct and stylistically awful, which is fine, but do not leave it that way.
4. Run both tools side by side for a while. Coexistence is supported; a big-bang cutover is not necessary.
5. Only then decide whether to move anything else.

If step two produces a diff you cannot explain, stop. An unexplained diff during an IaC migration is how resources get destroyed and recreated.

Common mistakes

• Switching tools to solve a licensing concern when OpenTofu solves it for free.
• Picking the tool the loudest engineer prefers rather than the one the whole team can maintain.
• Adopting CDKTF in 2026. It was archived in December 2025 and will not receive fixes.
• Treating a converted program as finished code rather than a starting point.
• Assuming Terraform state is safe because values are marked sensitive. Encrypt the backend.
• Migrating production first because it is the stack that matters most. It is the stack that matters most.
• Running two tools against the same resources without a clear ownership boundary.

Best practices whichever lane you pick

• Remote state with locking and encryption, from the first commit. Never local state for anything shared.
• Small stacks with clear boundaries. One enormous state file is the most common self-inflicted wound in this field.
• Plan or preview output in every pull request, and require a human to read it.
• Pin provider versions. Floating versions turn an unrelated deploy into an incident.
• Policy as code before you need it, not after the audit.
• Document the decision. Six months from now nobody will remember why, and the reasoning is worth more than the choice.

Frequently asked questions

Is Pulumi better than Terraform?

Neither is better in the abstract. Pulumi suits teams of software engineers building infrastructure with real logic in it. Terraform and OpenTofu suit teams provisioning a defined estate where constraint and readability matter more than expressiveness.

Can I use Terraform providers with Pulumi?

Yes. Pulumi bridges Terraform providers, either pre-built in its registry or generated on demand, and can consume Terraform modules directly. Bridged providers can lag upstream releases, so verify coverage for anything recent.

Should I move from Terraform to OpenTofu?

If licensing is your concern, it is close to free: same HCL, same providers, same state. Back up state, run tofu init and tofu plan, and confirm no changes. If you rely on Stacks, Sentinel, or HCP Terraform, staying put is reasonable.

What happened to CDKTF?

HashiCorp deprecated it on 10 December 2025 and archived the repository. It gets no further fixes or compatibility updates. If you want general-purpose languages for infrastructure now, Pulumi is the maintained option.

Does the BSL licence stop me using Terraform commercially?

Not for provisioning your own infrastructure, which is what almost everyone does. It restricts building a competing commercial IaC product on top of it. Check with your own legal team rather than a blog post if you are anywhere near that line.

Can Pulumi and Terraform coexist?

Yes, and for any migration of size that is the sensible approach. Pulumi can reference Terraform state and consume Terraform modules. The rule is that each resource has exactly one owner.

Wrapping up

The Pulumi vs Terraform decision is less about the tools than about who maintains your infrastructure and what it has to do. If your estate is a known set of environments maintained by people with mixed backgrounds, a constrained configuration language is a feature and OpenTofu gives you that without licensing anxiety. If you are building a platform, with branching logic and abstractions other teams consume, a real language earns its complexity and Pulumi is the maintained way to get one.

What has genuinely changed is that the lanes are no longer walled off. Pulumi speaks HCL and reads Terraform state. OpenTofu is a drop-in for Terraform. Converting between them is tooling rather than a rewrite. That makes the decision less permanent than it feels, which is a good reason to pick deliberately and a bad reason to agonise.

Pick the lane your team can still maintain when the person who chose it has moved on. Then write down why.

Need help choosing or migrating?

I work with teams on infrastructure as code decisions and the migrations that follow — Terraform to OpenTofu, Terraform to Pulumi, splitting up state files that grew too large, and wiring the whole thing into CI with policy checks that actually run.

If you would rather talk it through with someone who has untangled a few of these, you can hire me on Upwork.

The post Pulumi vs Terraform: Choosing a Lane appeared first on John Nessime.