Nothing about the database had changed. Same engine, same instance class, same subnet group. What changed was the resource’s name inside the module. The author had tidied up a resource "aws_db_instance" "this" into something more descriptive, shipped it as a minor release, and moved on. Terraform saw a resource at an address that no longer existed and a resource at an address that did not exist before. Its default reading of that is destroy and create.

That is the real cost of a module, and it is invisible on day one. The first apply is always fine. Modules feel great the week you adopt them. The bill arrives at the third upgrade, when the person who wrote it has moved teams and a dozen repositories consume it.

This post is about judging that cost before you pay it. What separates reusable Terraform modules that are genuinely worth adopting from the ones your team quietly pins at an old version and refuses to touch, how to make your own modules land on the right side of that line, and what to do when a plan comes back looking like the one above.

The bill for a module arrives at upgrade time

Terraform correlates state to configuration by address. Not by tags, not by cloud resource ID, not by anything the provider knows. Just the address: module.database.aws_db_instance.this. Change any part of that path and, unless you say otherwise, Terraform concludes the old object should go and a new one should take its place.

For a security group that is an annoyance. For a database, an object store, or anything holding data, it is an incident with a retro attached.

The fix has existed for a long time and it is genuinely good: the moved block. A module author who renames something declares the rename in code, ships it with the release, and every consumer’s next plan updates state instead of destroying infrastructure.

# Shipped inside the module, in the same release as the rename.
# Terraform rewrites the state address instead of planning a replacement.

moved {
  from = aws_db_instance.this
  to   = aws_db_instance.primary
}

The block is declarative and it chains. If a resource moves in one release and moves again two releases later, someone jumping several versions at once still lands correctly, because Terraform resolves the whole chain. That is why a module that keeps its moved blocks around is doing you a favour, and why one that strips them out after a release or two is quietly making every skipped upgrade riskier.

So here is the whole thesis, and everything below is elaboration: a module is worth reusing if the next version bump is cheap. Everything else is secondary.

Five signals that reusable Terraform modules are worth adopting

These are the things I actually check before letting a module into a codebase. None of them take more than a few minutes. All of them predict the upgrade experience better than star counts do.

1. It keeps state addresses stable, or tells you when it does not

Open the repository. Search for moved. Read the changelog for the last major release and see whether renames are called out.

A module with no moved blocks and no changelog is not necessarily bad code. It just means every internal refactor becomes your problem, discovered in a plan output, usually under time pressure. A module that says “v3.0 renames the launch template resource, here are the moved blocks” is telling you it has thought about consumers.

2. The input surface is narrow and typed

Count the variables. If a module needs forty inputs to cover its use cases, the abstraction was drawn in the wrong place. Usually that means two patterns got merged into one module because they looked similar from a distance.

Width is not the only tell. Look at how the inputs are typed. A module that takes twelve loose strings is very different from one that takes three typed objects with sensible defaults baked in.

variable "logging" {
  description = "Access log settings. Set to null to disable logging."
  type = object({
    bucket         = string
    prefix         = optional(string, "logs/")
    retention_days = optional(number, 30)
  })
  default = null
}

variable "environment" {
  description = "Deployment environment."
  type        = string
  nullable    = false

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be one of dev, staging or prod."
  }
}

Three things are doing work there. optional() with a default means a caller who wants the simple case writes two lines instead of six. nullable = false means a caller cannot pass null and get a confusing failure three resources deep. The validation block means a typo fails at plan time with a sentence a human wrote, rather than as a provider error about an invalid parameter.

That last one matters more than it looks. Every validation rule a module author writes is an error message you do not have to debug.

3. You can pin it, and the pin means something

This trips people up because the two common source types behave differently.

# Registry source. The version argument applies here.
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"   # allows any 5.x, refuses 6.0
}

# Git source. There is no version argument. Pin with ref.
module "internal_network" {
  source = "git::https://git.example.com/infra/tf-network.git?ref=v2.3.1"
}

The ~> operator only allows the rightmost specified component to increase. Written against a major and minor, as above, it accepts patch and minor releases and blocks the next major. Written as ~> 5.0.1 it accepts patches only. Which one you want depends entirely on whether you trust the author’s reading of semantic versioning, and the honest answer for third-party modules is usually “not enough to auto-accept minors in production”.

For Git sources, pin to a tag, not a branch. A branch reference resolves to whatever the tip happens to be at init time, which turns your infrastructure into a moving target and makes a plan from two weeks ago unreproducible.

4. It has tests you can run without a cloud account

Terraform’s built-in test framework uses .tftest.hcl files and the same HCL you already write. The critical detail for module authors is that a run block set to command = plan creates nothing. No credentials, no resources, no teardown, no cost. It just checks that the configuration produces what you claimed.

# tests/validation.tftest.hcl
# Runs on plan only. Nothing is created and nothing is billed.

run "rejects_unknown_environment" {
  command = plan

  variables {
    environment = "produciton"
  }

  expect_failures = [var.environment]
}

Negative tests like that are the cheapest quality signal in the whole ecosystem. They prove the validation rules actually fire, which is the difference between a rule that catches typos and a rule that was written once and silently broken later.

OpenTofu runs the same format through tofu test, and additionally recognises a .tofutest.hcl extension for tests that should only run there. Useful if you maintain modules that target both tools.

5. Somebody answers issues

Least technical, most predictive. Open the issue tracker, sort by recently updated, and look at whether maintainers reply. A module with a hundred open issues and active triage is in far better shape than one with six open issues and no comments since a provider major release.

Provider major versions are the stress test. When the AWS or Azure provider ships a breaking change, every module has to react. Modules without a maintainer simply stop being usable, and you find out when init refuses to resolve a version constraint.

Three kinds of module, and which ones actually pay off

Not everything that can be a module should be. It helps to name the three shapes you see in the wild.

The thin wrapper

One resource, a handful of variables passed straight through, maybe a tag merge. This is the most common module and usually the least valuable. It adds a layer of indirection, a version to track and a state address prefix, in exchange for saving four lines.

There is one case where it earns its place: when the wrapper is enforcing something, like mandatory encryption, a naming convention, or a tag schema your finance team depends on. Then it is not a wrapper, it is a policy with a friendly interface. If it enforces nothing, inline the resource and move on.

The composition module

Five to fifteen resources that only make sense together. A VPC with its subnets, route tables and gateways. A service with its task definition, target group, listener rule and log group. This is where modules earn back their cost, because the value is in the wiring, and the wiring is exactly what people get wrong when they copy and paste.

Most modules worth reusing live here.

The landing zone

An entire environment behind one module call. Accounts, networking, IAM baseline, logging, the lot. These are genuinely powerful and genuinely expensive. They only work when a team owns them, tests them, and has a release process. A landing zone module maintained by one person as a side project becomes the single scariest thing in the repository, because nobody can upgrade it and nobody can replace it either.

Public modules or your own?

The case for public modules is stronger than the “we have specific needs” crowd usually admits. The community AWS collections in particular have absorbed years of edge cases you have not thought of yet: the ordering constraints, the optional sub-resources, the arguments that only apply in certain regions. Reimplementing that from the provider docs is a real project, not an afternoon.

The case against is equally real. Popular public modules are wide by necessity, because they serve everyone. That width shows up as a large input surface, a lot of conditional logic, and majors that shuffle internals. You inherit an upgrade cadence you do not control.

My rough split: use public modules for undifferentiated infrastructure where the community has clearly solved it, and write your own for anything encoding a decision your organisation made. A VPC is undifferentiated. Your service deployment pattern is not.

There is a licensing dimension worth knowing about. Terraform moved from an open source licence to the source-available Business Source License, which is what prompted the OpenTofu fork now under the Linux Foundation. Module code itself is unaffected by that change, and the two tools read the same HCL, so modules generally work with either. OpenTofu runs its own registry rather than HashiCorp’s. If your organisation has strict open source licensing requirements, that distinction is the one to escalate, and it is a legal question rather than a technical one.

For sharing modules internally, a private registry gives you discovery and a version list in one place. HCP Terraform includes one, and the run platforms built around Terraform and OpenTofu, such as Spacelift, Scalr and env0, all offer module catalogs with policy enforcement attached. You can also get most of the benefit from tagged Git repositories on GitHub or GitLab and a README, which is where I would start before paying for anything. If you self-host your Git server and CI runners, they need somewhere to live, and a small VPS from a provider like InterServer, Hetzner or DigitalOcean handles module CI comfortably, since the plans are cheap and the tests never leave the plan phase.

Making your own modules worth reusing

If you are on the authoring side, this is the order I would do things in.

1. Write it inline first, twice. Do not modularise a pattern you have seen once. The second copy tells you which parts vary and which parts are actually fixed. That is the module’s interface, discovered rather than guessed.
2. Extract with moved blocks in the same commit. Moving a resource into a module changes its address. If the resource holds data, that is the difference between a refactor and an outage.
3. Type the inputs properly. Objects with optional() defaults over flat strings, nullable = false where null makes no sense, validation blocks with error messages that name the fix.
4. Prefer for_each over count for anything collection-shaped. With count, removing an element from the middle of a list shifts every index after it, and Terraform plans a rebuild of resources you never touched. Keyed addresses do not shift.
5. Generate the docs. terraform-docs reads your variables and outputs and writes the README table. Wire it into CI so the docs cannot drift from the code, because hand-written input tables are always out of date.
6. Lint and scan on every PR. terraform fmt -check, terraform validate, tflint for provider-specific mistakes, and a policy scanner such as Checkov or Trivy for the security defaults. All of these run in seconds on a plain GitHub Actions or GitLab CI runner.
7. Tag releases, and mean it. A major for anything that changes an address or removes an input. A minor for new optional inputs. Consumers pin against your tags, so the tag is a promise.

One thing I would skip early on: building a full integration test suite that applies real infrastructure. It is slow, it costs money, and it needs credentials in CI. Plan-mode tests plus a policy scanner catch most of what actually breaks. Add applied tests later, for the two or three modules where the wiring is genuinely subtle.

Troubleshooting the plans that scare you

“Forces replacement” after a version bump

Before you argue with it, find out what it wants to delete. Save the plan and read it as JSON rather than scrolling terminal output.

terraform init -upgrade
terraform plan -out=tfplan

# List every address the plan intends to delete.
terraform show -json tfplan | jq -r '.resource_changes[]
  | select(.change.actions[] == "delete") | .address'

If the list is empty, the replacement was an argument change, not an address change, and you should read the diff on that resource. If the list contains things you did not expect, compare the addresses against the new module version’s source. A rename with no moved block is the usual answer, and you can write the moved block yourself in your root configuration as a workaround.

A module with a provider block inside it

Modules should declare which providers they need, not configure them. A module that contains a configured provider block rather than just a required_providers entry limits what callers can do with it, including using for_each or count on the module call. It also makes the module harder to destroy cleanly. If you hit that, the fix is on the module side: move provider configuration up to the root and pass configuration aliases in.

Changes to the module source are not picked up

Remote modules are downloaded into the working directory during init. Changing a ref or a version constraint in your configuration does not re-download anything on its own. Run terraform init -upgrade. If your CI caches the module directory between runs, that cache is the first place to look when a pipeline plans against code you are sure you deleted.

Registry version list looks stale

A registry’s list of versions is a synced view of your Git tags, not an independent database. Reconnecting a VCS integration, moving repositories, or pushing many tags at once can leave the registry out of step with the repository. Treat the tags as the source of truth and check the registry’s view after any change to that integration.

Common mistakes

• Pinning to a branch. ?ref=main is not a pin. It is an unannounced deployment scheduled for whenever someone next runs init.
• Modularising after one use. You end up encoding the first use case’s assumptions as the abstraction, then bolting flags onto it for every case after.
• Copying environments instead of parameterising them. Three near-identical directories drift within a quarter, and the fix that went into dev never reaches prod.
• Passing every provider argument straight through. If a module’s variables are a mirror of the resource schema, it is not an abstraction, it is a rename with extra steps.
• Deleting moved blocks too early. They only cost you a few lines. Removing them is safe only when you are certain every consumer has applied past that version, which you can rarely be sure of for a public module.
• Applying a plan you did not read. Nearly every module horror story ends with someone approving a plan whose destroy count they never looked at.

How I decide whether to adopt a module

Roughly ten minutes, in this order:

1. Count the input variables. Over about thirty, I assume the abstraction is wrong and look for a narrower alternative.
2. Grep for moved. Its presence tells me the author thinks about consumers.
3. Read the last major release notes. Are breaking changes explained, with an upgrade path?
4. Check for a tests directory and whether CI runs on pull requests.
5. Look at the issue tracker’s recent activity, not its total count.
6. Adopt it in one non-production configuration first, then bump it once deliberately and watch the plan. That single bump tells you more than the other five steps combined.

And a standing rule regardless of the module: never let a plan with a non-zero destroy count through review without someone saying out loud what is being destroyed and why.

Frequently asked questions

How many input variables is too many for a Terraform module?

There is no hard limit, but a wide input surface is a symptom rather than a style choice. If covering your use cases needs dozens of inputs, the variation between callers is probably larger than what they share, which means you have found two patterns rather than one. Splitting into two narrower modules usually reads better and upgrades more safely than adding a twelfth boolean flag.

Should I use public modules or write my own?

Both, for different things. Public modules are a good trade for undifferentiated infrastructure where the community has already absorbed the edge cases. Write your own for anything that encodes a decision specific to your organisation, because that is the part nobody else will maintain for you and the part that changes on your schedule rather than someone else’s.

Do Terraform modules work with OpenTofu?

Generally yes. A module is just a directory of HCL, and both tools read the same language and use compatible provider plugins, so registry modules, private modules and Git-sourced modules usually work unchanged. Where you can hit friction is version constraints and features that have diverged between the two projects since the fork. Test one workspace before committing an estate to a migration.

How do I pin a module version from a Git source?

The version argument only applies to registry sources. For Git, append a ref query parameter to the source URL and point it at a tag or a commit SHA. Branch names resolve to whatever the tip is at init time, which defeats the purpose of pinning entirely.

What happens if a module removes a resource in a new version?

Terraform plans to destroy it, because the resource is in state and no longer in configuration. Sometimes that is exactly what the author intended. Sometimes it means functionality moved somewhere you have not adopted yet. Read the release notes before applying, and if the object holds data, take a snapshot first regardless of what the notes say.

Can I test a Terraform module without cloud credentials?

Yes, for a useful subset. Run blocks set to command = plan evaluate the configuration without creating anything, which is enough to assert on computed values, defaults and validation failures. Testing behaviour that only exists after creation still needs an apply and therefore real credentials, which is why most module suites are mostly plan tests with a small number of applied ones.

Do I need a private registry to share modules internally?

No. Tagged Git repositories and a source URL cover the mechanics completely. A registry buys you discovery, a browsable version list and a place to attach policy, which starts mattering once enough teams are publishing that people cannot find what already exists. Below that point it is overhead you do not need yet.

Conclusion

Most advice about reusable Terraform modules focuses on the writing: structure, naming, README quality. That is the easy part, and it is not where things go wrong.

The modules that are actually worth reusing are the ones you can still upgrade in two years. Stable state addresses, a narrow typed interface, a pin that means something, tests that run on plan, and a maintainer who replies. If you only remember one thing, remember what a module is really promising you: not that the first apply will work, but that the next version bump will not wake anyone up.

Need a second pair of eyes on your module library?

I work with teams on the boring, high-consequence parts of Terraform and OpenTofu estates. Things I help with regularly:

• Auditing an existing module library and saying plainly which modules earn their keep and which should be inlined back into root configurations
• Adding moved blocks and upgrade notes to in-house modules so version bumps stop being incidents
• Narrowing wide module interfaces with typed objects, optional() defaults and validation rules that fail with useful messages
• Setting up module CI: fmt, validate, tflint, policy scanning and generated docs on every pull request
• Writing terraform test suites that run on plan, including negative tests that prove your validation rules actually fire
• Planning a Terraform to OpenTofu migration across a module estate, or working out whether it is worth doing at all

Send me a module’s variables.tf and the plan output from your last version bump. That pair is usually enough for me to tell you where the trouble is.

The post Terraform Modules Worth Reusing (And the Ones That Cost You) appeared first on John Nessime.

That is the failure that actually bites when you provision servers by hand for a handful of clients. It is invisible for months. Nothing alerts. Nothing breaks. Then a disk dies, or a client asks for a second region, or you want to hand the whole thing over, and the gap between “the server works” and “the server is reproducible” turns into a weekend.

This post covers how to automate server provisioning for new clients in a way that survives that moment: what to put in the image, what to put in first boot, what to leave to configuration management, how to keep one client’s credentials from ever touching another client’s box, and how to prove a build is finished instead of assuming it.

The four failure families worth designing against

Automation is not the goal. Automation is what you reach for once you have decided which of these you refuse to keep living with. Pick the ones that apply to you and let them shape the build, because trying to solve all four at once is how provisioning projects stall.

1. You cannot rebuild it

The test is blunt: destroy the machine and rebuild it from your repository, with no shell access to the old one. If you cannot do that, you do not have a server, you have an artefact. Every hand-run apt install, every file edited over SSH at 1am, every “I’ll document that later” is a piece of the machine that exists nowhere but the machine.

This is the one that stays invisible longest. A hand-built box can run happily for two years. The cost lands entirely in one bad afternoon.

2. Every client is a slightly different server

Client A got fail2ban. Client B got it two months later with different jail settings. Client C never got it because that build happened on a Friday. Now a CVE lands in a package you installed on four of six boxes and you have to go look at each one to find out.

Drift is not just untidy. It destroys your ability to reason about the fleet as a fleet. Any statement that starts “all our clients have…” becomes a guess.

3. One credential, too much blast radius

This one is specific to consulting and it is the one people get wrong most often. If the same SSH key opens every client’s server, and the same API token provisions every client’s infrastructure, then a compromise of your laptop is a compromise of everybody. You are a shared dependency with none of the controls a shared dependency is supposed to have.

Worse, it is contractually awkward. Plenty of client agreements have language about access control that a single reused key quietly violates.

4. The handover you cannot perform

Engagements end. When one does, the client should be able to take a repository, run it, and get their infrastructure. If the only way to reproduce their environment is to hire you again, that is not leverage, it is a liability. It also makes you slower to say yes to the next client, because your capacity is tied up in machines only you understand.

How to automate server provisioning for new clients

The stack that holds up in practice has four layers, and the important part is the boundary between them. Most broken provisioning setups I see are broken because someone pushed work into the wrong layer.

Layer 1: the machine itself

Terraform or OpenTofu talks to your provider’s API and creates the server, the firewall rules, the volumes and the DNS records. Whether you run a VPS at InterServer, Hetzner, Vultr or DigitalOcean, or full cloud on AWS, the shape is the same: a declarative file that says what should exist.

Worth knowing before you commit: HashiCorp moved Terraform from the MPL to the Business Source License, which is source-available rather than open source, and the community forked the last MPL release as OpenTofu under the Linux Foundation. For an independent engineer building client infrastructure, either is workable. Terraform has the larger provider and module ecosystem and the commercial backing. OpenTofu gives you a licence that will not be renegotiated under you, which matters more when the code you write ends up in a client’s repository and out of your control. The HCL, provider protocol and state model are shared, so the switching cost is low in both directions. I lean OpenTofu for client work for the licence certainty, and that is a judgement call, not a technical verdict.

Keep state remote and locked from day one, and keep one state file per client. Shared state is the fastest way to turn a mistake on one engagement into an outage on another.

Layer 2: first boot with cloud-init

cloud-init runs once, on first boot, before you have any way in. Its job is narrow: create the account you will connect as, install its key, lock down password and root login, and get out of the way. Everything else belongs in the next layer.

#cloud-config
hostname: web01
fqdn: web01.acme.internal
timezone: Etc/UTC

users:
  - name: deploy
    groups: [sudo]
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nz... provisioning key for this client only

# Refuse passwords and direct root logins from the very first boot,
# so there is no window where the box is reachable with weaker settings.
ssh_pwauth: false
disable_root: true

package_update: true
packages:
  - python3
  - sudo

Two deliberate omissions. There is no package_upgrade: true, because a full upgrade at first boot can run for minutes, holds the package lock, and is the single most common cause of the race described in the troubleshooting section below. Do the upgrade in Ansible where you can see it fail. And there is no application setup, because anything in runcmd is a script you will never test again.

python3 is there so Ansible has an interpreter to work with. Some images ship it already; installing it is cheap insurance.

Validate the file before it ever reaches a provider, because a YAML error in user-data fails silently and you find out by not being able to log in:

cloud-init schema --config-file user-data.yaml

Layer 3: convergence with Ansible

This is where the actual build lives: packages, users, firewall, TLS, web server, database, backup agent, monitoring agent, log shipping. Ansible is agentless and runs over SSH, which fits consulting well because you are not asking a client to accept a permanent daemon on their box.

The layout that scales across clients separates the shared build from the per-client differences:

infra/
  roles/                  # shared, versioned, the same for everyone
    base/
    hardening/
    web/
    backup/
  clients/
    acme/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
    globex/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
  site.yml

Roles are the product. Client directories are configuration. If you find yourself copying a role into a client directory and editing it, that is a variable you have not extracted yet.

Hand the inventory over from Terraform rather than maintaining it by hand. The simplest reliable version reads the outputs as JSON:

cd clients/acme/terraform

tofu output -json client_hosts 
  | jq -r '.value[] | "(.name) ansible_host=(.ipv4)"' 
  > ../inventory/hosts.ini

There are richer options, including inventory plugins that read Terraform state directly, but a generated file has a property worth keeping: you can read it, diff it, and commit it. When a provisioning run targets the wrong host, that diff is how you find out.

Layer 4: the verification gate

This is the layer almost everyone skips, and it is the one that turns provisioning from a script into a process. A playbook that finishes without errors has told you that no task raised an exception. It has not told you the server works.

Write a separate play whose only job is to assert the finished state, and run it as the last step of every provision:

- name: Verify the host is genuinely ready
  hosts: all
  gather_facts: true
  tasks:
    # raw, not command: this may run before Python is guaranteed present
    - name: Wait for cloud-init to finish
      ansible.builtin.raw: cloud-init status --wait
      changed_when: false

    - name: Collect service state
      ansible.builtin.service_facts:

    - name: Core services are running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'
          - ansible_facts.services[ssh_unit].state == 'running'
        fail_msg: "Core services are not up on {{ inventory_hostname }}"

    - name: Site answers over TLS
      ansible.builtin.uri:
        url: "https://{{ client_domain }}/"
        status_code: 200
      delegate_to: localhost

Note ssh_unit as a variable rather than a literal: the SSH unit is ssh.service on Debian and Ubuntu but sshd.service on RHEL-family systems. Hard-coding it is how a verification play passes on one distro and fails confusingly on another.

The point of the gate is not that it catches exotic bugs. It is that it catches the ordinary ones: the service that installed but did not enable, the certificate that issued but did not get loaded, the firewall rule that opened the wrong port.

Keeping clients apart: secrets and access

Generate a fresh SSH keypair per client engagement. It costs you thirty seconds and it means revoking access to one client’s infrastructure is a single, contained action.

For secrets in the repository you have two credible options. Ansible Vault is built in and needs nothing extra, but it encrypts whole files, so your diffs become meaningless blobs and rotating a key across several clients is tedious. SOPS encrypts values while leaving keys readable, which means a code review can still see that a variable changed without seeing what it changed to, and it supports age, PGP and the cloud KMS services. For a repo that a client might eventually inherit, that readability is worth the extra tool.

# .sops.yaml at the repo root: different key per client
creation_rules:
  - path_regex: clients/acme/secrets/.*.ya?ml$
    age: age1acme...
  - path_regex: clients/globex/secrets/.*.ya?ml$
    age: age1globex...

Separate keys per client is the whole point. One shared key and you are back to a single blast radius with extra ceremony.

For handing the initial credentials to a client, use something with an audit trail and an expiry. A shared vault item in 1Password or Bitwarden is fine. Email and chat are not, because the secret then lives in two message histories neither of you controls.

If you want ongoing access without exposing SSH to the internet at all, an overlay network like Tailscale installed by the base role is a clean way to do it. The provisioning path still uses a public address for the first run, then the firewall closes it.

Troubleshooting a provisioning run

Ansible connects before cloud-init has finished

The most common failure by a wide margin. SSH comes up before cloud-init is done, Ansible connects, and then the first package task fails because cloud-init still holds the package manager lock. The error points at apt or dnf, which sends you looking in entirely the wrong place.

Fix it by blocking explicitly at the top of the first play, as in the verification snippet above. Check the state manually with:

cloud-init status --wait --long
sudo grep -i Traceback /var/log/cloud-init.log

One caveat that catches people: never put cloud-init status --wait inside a runcmd or bootcmd block. It will deadlock waiting for the process that is running it.

cloud-init appears to have done nothing

Two usual causes. Either the user-data was malformed, in which case cloud-init skipped it rather than erroring loudly, or the modules already ran on a previous boot and the semaphore files in /var/lib/cloud/instance/sem/ are preventing a rerun.

# what actually ran, and how long each stage took
cloud-init analyze show

# the human-readable output of scripts and package installs
sudo less /var/log/cloud-init-output.log

# on a throwaway test box only: wipe state and start over
sudo cloud-init clean --logs --reboot

Never run cloud-init clean on a production host. It removes cached state and the following boot will behave as a first boot.

Host key verification failed on a rebuilt server

Rebuild a machine on the same IP and every client with the old key in known_hosts refuses to connect. The tempting fix is to disable host key checking in ansible.cfg. Do not. That turns a real integrity check into a permanent no-op across every client you manage.

Remove the stale entry and pin the new one as part of the provisioning flow:

ssh-keygen -R 203.0.113.10
ssh-keyscan -H 203.0.113.10 >> ~/.ssh/known_hosts

Better still, if your provider exposes the host key fingerprint through its API or console output, compare it before you accept. That closes the trust-on-first-use gap properly.

The playbook reports changes on every single run

Almost always a shell or command task with no creates, changed_when or module equivalent. Ansible cannot know whether a raw command changed anything, so it assumes yes. Real changed counts are what let you spot drift, and a playbook that always reports twelve changes tells you nothing.

Run against an already-provisioned host in check mode. Anything that reports a change is either genuine drift or a task you need to fix:

ansible-playbook -i clients/acme/inventory/hosts.ini site.yml --check --diff

Common mistakes

• Putting the application build in runcmd. It runs once, never gets tested, and is invisible to every later change.
• One shared SSH key and one shared API token across all clients. Convenient right up to the moment it is not.
• Copying a role into a client directory to make one small change, instead of extracting a variable.
• Treating “the playbook exited zero” as proof the server is ready. It is proof no task threw.
• Local Terraform state on your laptop. It is a single point of failure that also cannot be handed over.
• Provisioning the server but not the DNS, TLS, monitoring and backups. A box nobody watches and nobody can restore is not finished.
• Automating the build and then fixing production by hand. One SSH session and you are back to a snowflake.

Best practices

• Rebuild before you need to. Destroy and rebuild a client’s staging host on a schedule. It is the only honest test of reproducibility.
• Version your roles and pin them per client. A shared role that everyone tracks on main means one commit changes six clients at once.
• Keep first boot minimal. If it can wait for Ansible, it should.
• Make monitoring part of provisioning. Whether that is Prometheus you run yourself, Grafana Cloud, or a lighter uptime service, the agent goes on in the base role, not as a follow-up ticket.
• Same for backups. An offsite target such as Backblaze B2 configured at build time, plus a restore test, or you have a backup you have never proven.
• Separate credentials per client, always. Keys, tokens, encryption keys, state backends.
• Write the runbook as you build. The provisioning repo’s README is the handover document. If it is accurate, the engagement can end cleanly.

Frequently asked questions

Do I need Terraform if I only manage a few VPS instances?

Not necessarily. If you are creating three servers a year through a provider’s control panel, the honest answer is that Terraform adds a state file to look after for very little return. Start with cloud-init and Ansible, which give you most of the reproducibility. Add Terraform or OpenTofu when the infrastructure around the server, such as firewalls, volumes, load balancers and DNS records, starts to be the part you keep getting wrong.

Should I build custom images instead of provisioning at boot?

Golden images cut boot time and remove dependency on package mirrors being up during a build. They also add a pipeline you have to maintain and rebuild for every security update. For client work at small scale, provisioning from a stock distribution image is usually the better trade, because the build stays readable and there is no stale image to forget about. Images earn their cost when you are launching machines often enough that boot time matters.

Terraform or OpenTofu for client projects?

Both work, and the configuration language is shared, so this is not a lock-in decision in the way it looks. Terraform has the deeper ecosystem and the commercial support path. OpenTofu is under the Linux Foundation with a licence that is unlikely to change, which is the stronger argument when the code will live in a client repository long after you have moved on. Ask the client whether they have a standard before you pick.

How do I automate server provisioning without exposing SSH to the internet?

The first run generally needs a reachable address, so scope the firewall rule to your own source IP for the duration of provisioning and close it in the same run. After that, reach the host through a bastion or an overlay network installed during the base role. Provider firewalls and host firewalls should both be in the automation, because relying on only one of them means a provider console change can silently open the box.

How do I stop configuration drift once the server is live?

Run the playbook in check mode on a schedule and look at what it reports. Any change on a host you have not touched is drift, and it is far cheaper to find it that way than during an incident. This only works if your tasks are properly idempotent, which is why the always-reports-changed problem above matters more than it looks.

What belongs in cloud-init versus Ansible?

cloud-init handles what must exist before you can connect: hostname, the login account, its SSH key, disabled password authentication, and a Python interpreter. Ansible handles everything you will ever want to change again. The dividing line is repeatability. cloud-init runs once and you cannot re-run it safely on a live host, so anything you might revisit belongs on the Ansible side.

How long should provisioning a new client server take?

Wall-clock time is the wrong measure. The useful measure is how much of it needs you. A build that takes twenty minutes but runs unattended from one command is in far better shape than a five-minute build with four manual steps, because the manual steps are where the divergence between clients gets introduced.

The one thing worth remembering

When you automate server provisioning for new clients, the deliverable is not the script. It is the ability to answer “can you rebuild this?” with yes, on any day, without opening a shell on the existing machine.

Everything else follows from that. Reproducibility gives you consistency across the fleet, because there is one definition. It gives you a clean handover, because the repository is the environment. It gives you the ability to take the next client on without dreading the build. And it costs the same afternoon that a single unplanned rebuild would have cost you anyway, just spent on your own schedule instead of at 2am.

Start with one client. Get their staging box rebuilding from scratch. Then make the second client’s build reuse the roles from the first. That is the whole path.

Need a provisioning setup you can actually hand over?

Most of the provisioning work I take on is not greenfield. It is a set of hand-built servers that work fine and nobody can reproduce. Things I can help with:

• Turning existing hand-built client servers into an Ansible role set, without a rebuild and without downtime
• Building a client onboarding repository: Terraform or OpenTofu, cloud-init, roles, per-client inventory and secrets
• Splitting shared credentials into per-client keys, tokens and state backends, with a revocation path that works
• Writing the verification playbook that decides whether a build is finished, and wiring it into CI
• Fixing idempotency so check-mode runs become a usable drift report instead of noise
• Preparing an environment for handover: runbook, access review, and a documented rebuild that the client can run themselves

If you want a second opinion on something specific, send the cloud-config, the playbook output, or a check-mode diff you cannot explain, and I will tell you what I see.

The post Automating Server Provisioning for New Clients Without Building a Snowflake Farm appeared first on John Nessime.

The Pulumi vs Terraform question got harder in the last two years, not easier. The licence changed, the community forked, IBM bought HashiCorp, HashiCorp killed its own general-purpose-language option, and Pulumi started speaking HCL. A lot of the comparison articles you will find were written before any of that happened.

This is an attempt at an honest read of where things actually stand, what the real trade-off is once you strip away the marketing, and how to pick a lane you will not regret in eighteen months.

What actually changed

A short timeline, because the current landscape only makes sense against it.

• August 2023 — HashiCorp relicensed Terraform from MPL 2.0 to the Business Source License. Source-available, free for provisioning your own infrastructure, but you cannot build a competing commercial product on it.
• January 2024 — OpenTofu shipped 1.6.0, a fork of the last MPL-licensed code, governed by the Linux Foundation.
• February 2025 — IBM completed its acquisition of HashiCorp, valued at around $6.4 billion.
• December 2025 — HashiCorp deprecated CDK for Terraform and archived the repository. The official way to write Terraform in a general-purpose language no longer exists.
• Also December 2025 — Pulumi announced native HCL support and the ability to work with Terraform and OpenTofu state.

Those last two matter more than the licence drama, and they point in opposite directions. HashiCorp closed the door on writing Terraform in a real language. Pulumi opened the door to writing Pulumi in HCL. Whatever else you conclude, “HCL versus real languages” is no longer a clean line between two products.

The real axis: a configuration language or a programming language

Strip out the licensing and the corporate ownership and one genuine difference remains. HCL is a purpose-built configuration language that describes desired state. Pulumi runs an actual program that constructs a resource graph, and then applies it.

Here is the same trivial thing in both. Terraform:

resource "aws_s3_bucket" "logs" {
  for_each = toset(var.environments)
  bucket   = "acme-logs-${each.key}"

  tags = {
    Environment = each.key
    ManagedBy   = "terraform"
  }
}

And Pulumi, in TypeScript:

import * as aws from "@pulumi/aws";

const environments = ["dev", "staging", "prod"];

const buckets = environments.map(env =>
  new aws.s3.Bucket(`logs-${env}`, {
    bucket: `acme-logs-${env}`,
    tags: {
      Environment: env,
      ManagedBy: "pulumi",
    },
  })
);

For something this small the difference is aesthetic. It stops being aesthetic around the third nested conditional.

What you actually gain with a general-purpose language

• Real abstractions. Classes, functions, inheritance. A Pulumi component that encapsulates “our standard service” is just a class, and it composes the way code composes.
• Real testing. Jest, pytest, Go’s testing package. You can unit test infrastructure logic with the same tools and the same CI as the rest of your codebase.
• The package ecosystem. Need to parse a CSV of subnets, call an internal API, or do something genuinely awkward? It is an npm install rather than an external data source and a shell script.
• IDE support that works. Type checking, refactoring, jump-to-definition. HCL tooling has improved a lot but it is not the same thing.

What you lose, which comparison posts skip

This is the part worth reading twice, because it is where the regret comes from.

• Constraint is a feature. HCL is limited on purpose. When a language cannot express clever things, nobody writes clever things. Give a team a real language and someone will eventually build a metaprogramming layer that generates resource names from a database lookup, and reviewing that pull request will be nobody’s idea of a good afternoon.
• Readability across skill levels. A network engineer who has never written TypeScript can usually read HCL and tell you what it provisions. The reverse is not true. If your infrastructure is touched by people who are not primarily programmers, that matters enormously.
• The hiring pool. There are far more engineers with Terraform on their CV than Pulumi. That is not an argument about quality, it is an argument about how long it takes to backfill a role.
• A dependency on a runtime. Pulumi programs need Node, Python, or Go present and correct in every environment that runs them, including CI. It is one more thing that can drift.
• Asynchrony leaks into the model. Resource outputs are not plain values. They are futures, and you have to handle them as such. Every Pulumi team hits this on day two.

# This does not do what a newcomer expects. bucket.id is an Output,
# not a string, so you get an object reference in the name.
# bucket_name = f"logs-for-{bucket.id}"

# Correct: work inside apply, where the value is resolved.
bucket_name = bucket.id.apply(lambda bid: f"logs-for-{bid}")

None of these are dealbreakers. They are just the bill that arrives later, and it is worth reading before you sign.

State, providers, and the things that actually break

State management

Both tools keep a state file mapping your code to real resources, and both will ruin your afternoon if that file gets corrupted or out of sync.

Terraform and OpenTofu use pluggable backends: S3 with DynamoDB locking, Azure Blob, GCS, Consul, or HCP Terraform. OpenTofu added client-side state encryption, which Terraform does not have natively and which is a genuine advantage if your compliance people care about what is sitting in that bucket.

Pulumi defaults to Pulumi Cloud, which is free for individuals and paid for teams. It also supports self-managed backends, and you should know this because the default sometimes gets mistaken for a requirement:

# Self-managed state in S3, no Pulumi Cloud account involved
pulumi login s3://my-pulumi-state-bucket

# Or entirely local, useful for experiments
pulumi login --local

Pulumi encrypts secrets in state by default. With Terraform, values marked sensitive are redacted in output but still sit in plaintext in the state file, which surprises people who assumed otherwise.

The provider ecosystem, and a myth worth killing

You will read that Pulumi has a much smaller provider ecosystem. That was true once and is mostly no longer the point, because Pulumi bridges Terraform providers. If a Terraform provider exists, you can almost certainly use it from Pulumi, either pre-built in the Pulumi Registry or generated on demand.

Two caveats that are real:

• Bridged providers can lag upstream releases. If you need a resource type that shipped last week, check before committing.
• When something breaks deep in a bridged provider, you are debugging across a translation layer. That is a worse afternoon than debugging the provider directly.

Pulumi also ships native providers for Kubernetes and Azure, generated from the platform APIs, which give same-day coverage of new resources rather than waiting on a bridge.

Policy and governance

Terraform’s first-class answer is Sentinel, which is tied to HCP Terraform. OpenTofu users generally reach for OPA or Conftest. Pulumi has CrossGuard, where policies are written in the same languages as everything else.

If you are in a regulated environment and your auditors already know what Sentinel is, that is a heavier thumb on the scale than any language preference.

Licensing, since it is what forced the question

• Terraform — BSL 1.1. Source-available. Fine for provisioning your own infrastructure. Not fine for building a product that competes with HashiCorp. Each release converts to an open licence four years after publication, but only that release.
• OpenTofu — MPL 2.0, OSI-approved, Linux Foundation governance. No single vendor can relicense it.
• Pulumi — Apache 2.0 for the engine and CLI. Permissive, with a patent grant. Pulumi Corp monetises through Pulumi Cloud rather than through the licence.

For the overwhelming majority of teams, the BSL changes nothing about daily work. You are provisioning your own infrastructure and that is explicitly permitted. The concern is not today’s terms, it is that a single vendor demonstrated it can change them, and that vendor is now part of IBM.

If your objection to Terraform is licensing rather than language, the answer is OpenTofu, not Pulumi. Switching paradigms to solve a legal problem is an expensive way to do it.

Choosing a lane

Three coherent positions. Pick the one that matches your constraints rather than the one that sounds most modern.

Lane one: OpenTofu

The default for most teams already running Terraform who are uneasy about the licence. Migration is genuinely close to free: same HCL, same providers, same state format, same commands.

# Back up state first. Always.
terraform state pull > backup-$(date +%F).tfstate

# Then initialise with tofu against the same backend
tofu init
tofu plan

A clean plan with no proposed changes means you are done. That is the whole migration for most codebases.

Choose this if: you have existing Terraform, HCL is working for you, and you want open governance without retraining anyone.

Lane two: Terraform with HCP

Still the biggest ecosystem, the most documentation, the most Stack Overflow answers, the most people who already know it. Stacks and ephemeral resources are Terraform-only. Sentinel is Terraform-only. If you are deep in HCP Terraform and it works, the licence is an abstract concern rather than a practical one.

Choose this if: you are in a regulated industry, you need vendor support with a contract behind it, or your organisation is standardising on IBM and Red Hat tooling anyway.

Lane three: Pulumi

The strongest case for Pulumi is not “real languages are nicer”. It is when your infrastructure genuinely needs logic that HCL fights you on: dynamic topologies, infrastructure derived from application code, platform teams building abstractions for other teams to consume. If you are writing a self-service platform rather than a set of environments, a real language stops being a preference and starts being the right tool.

Choose this if: your team is primarily software engineers, your infrastructure has real branching logic, or you are building a platform product rather than provisioning a fixed estate.

Do not choose it because writing loops in HCL annoyed you once.

What migrating actually looks like

Terraform to OpenTofu is a swap. Terraform to Pulumi is a refactor, and anyone telling you otherwise is selling something.

Pulumi does provide real tooling for it:

# Convert HCL into a Pulumi program in your chosen language
pulumi convert --from terraform --language typescript

# Adopt the resources described by an existing state file
pulumi import --from terraform ./terraform.tfstate

Pulumi can also consume Terraform modules directly, so existing module investment is not automatically wasted. That is the single most useful fact for anyone considering a gradual move.

A sane sequence:

1. Pick one small, low-risk stack. Not the production VPC.
2. Convert it, import the state, and run a preview. The only acceptable result is no changes.
3. Refactor the generated code. Converted output is mechanically correct and stylistically awful, which is fine, but do not leave it that way.
4. Run both tools side by side for a while. Coexistence is supported; a big-bang cutover is not necessary.
5. Only then decide whether to move anything else.

If step two produces a diff you cannot explain, stop. An unexplained diff during an IaC migration is how resources get destroyed and recreated.

Common mistakes

• Switching tools to solve a licensing concern when OpenTofu solves it for free.
• Picking the tool the loudest engineer prefers rather than the one the whole team can maintain.
• Adopting CDKTF in 2026. It was archived in December 2025 and will not receive fixes.
• Treating a converted program as finished code rather than a starting point.
• Assuming Terraform state is safe because values are marked sensitive. Encrypt the backend.
• Migrating production first because it is the stack that matters most. It is the stack that matters most.
• Running two tools against the same resources without a clear ownership boundary.

Best practices whichever lane you pick

• Remote state with locking and encryption, from the first commit. Never local state for anything shared.
• Small stacks with clear boundaries. One enormous state file is the most common self-inflicted wound in this field.
• Plan or preview output in every pull request, and require a human to read it.
• Pin provider versions. Floating versions turn an unrelated deploy into an incident.
• Policy as code before you need it, not after the audit.
• Document the decision. Six months from now nobody will remember why, and the reasoning is worth more than the choice.

Frequently asked questions

Is Pulumi better than Terraform?

Neither is better in the abstract. Pulumi suits teams of software engineers building infrastructure with real logic in it. Terraform and OpenTofu suit teams provisioning a defined estate where constraint and readability matter more than expressiveness.

Can I use Terraform providers with Pulumi?

Yes. Pulumi bridges Terraform providers, either pre-built in its registry or generated on demand, and can consume Terraform modules directly. Bridged providers can lag upstream releases, so verify coverage for anything recent.

Should I move from Terraform to OpenTofu?

If licensing is your concern, it is close to free: same HCL, same providers, same state. Back up state, run tofu init and tofu plan, and confirm no changes. If you rely on Stacks, Sentinel, or HCP Terraform, staying put is reasonable.

What happened to CDKTF?

HashiCorp deprecated it on 10 December 2025 and archived the repository. It gets no further fixes or compatibility updates. If you want general-purpose languages for infrastructure now, Pulumi is the maintained option.

Does the BSL licence stop me using Terraform commercially?

Not for provisioning your own infrastructure, which is what almost everyone does. It restricts building a competing commercial IaC product on top of it. Check with your own legal team rather than a blog post if you are anywhere near that line.

Can Pulumi and Terraform coexist?

Yes, and for any migration of size that is the sensible approach. Pulumi can reference Terraform state and consume Terraform modules. The rule is that each resource has exactly one owner.

Wrapping up

The Pulumi vs Terraform decision is less about the tools than about who maintains your infrastructure and what it has to do. If your estate is a known set of environments maintained by people with mixed backgrounds, a constrained configuration language is a feature and OpenTofu gives you that without licensing anxiety. If you are building a platform, with branching logic and abstractions other teams consume, a real language earns its complexity and Pulumi is the maintained way to get one.

What has genuinely changed is that the lanes are no longer walled off. Pulumi speaks HCL and reads Terraform state. OpenTofu is a drop-in for Terraform. Converting between them is tooling rather than a rewrite. That makes the decision less permanent than it feels, which is a good reason to pick deliberately and a bad reason to agonise.

Pick the lane your team can still maintain when the person who chose it has moved on. Then write down why.

Need help choosing or migrating?

I work with teams on infrastructure as code decisions and the migrations that follow — Terraform to OpenTofu, Terraform to Pulumi, splitting up state files that grew too large, and wiring the whole thing into CI with policy checks that actually run.

If you would rather talk it through with someone who has untangled a few of these, you can hire me on Upwork.

The post Pulumi vs Terraform: Choosing a Lane appeared first on John Nessime.