None of that is a bug. AWS optimises the empty account for getting something running in ten minutes. The problem is that most accounts never get corrected afterwards, and eighteen months later there is production traffic sitting on top of decisions nobody made deliberately.

An AWS account baseline is the set of things you lay down before the first workload, so the account starts from a defensible posture instead of a blank console. This walks through the decisions that actually matter, why each one is the way it is, and what it costs. I have put a working reference implementation on GitHub — terraform-aws-baseline — so you can read the real Terraform rather than take my word for any of it.

What a baseline is, and what it is not

A baseline is not a landing zone in the Control Tower sense, and it is not a compliance framework. It does not make you SOC 2 compliant and nothing that fits in one repository will.

What it is: the boring foundational layer that everything else sits on. State storage, network segmentation, identity, and the guardrails that stop a careless afternoon becoming an incident. If you get these right once, every workload that lands afterwards inherits them for free.

The structure that works is three layers. A bootstrap stack that creates the state backend and runs exactly once. Reusable modules for the things every environment needs. Thin per-environment roots that compose those modules with different parameters — dev gets one NAT gateway and seven-day log retention, prod gets one per availability zone and a year.

Decision one: no long-lived AWS credentials, anywhere

Static access keys in CI are the single most common way AWS accounts get compromised. They live in a secrets store, they get copied into a second one, somebody echoes one into a build log, and nobody rotates them because rotation breaks things.

GitHub Actions can assume an AWS role using a short-lived OIDC token that GitHub signs and AWS verifies. There is no key to store, nothing to leak, and nothing to rotate.

The important part is the trust policy, and one word in it decides whether this is a security improvement or a vulnerability:

data "aws_iam_policy_document" "ci_assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    # StringEquals, not StringLike. This is the whole ballgame.
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values = [
        "repo:${var.github_org}/${var.github_repo}:ref:refs/heads/${var.github_branch}"
      ]
    }
  }
}

Swap StringEquals for StringLike and add an asterisk, and you have widened that trust from one branch of one repository to whatever the wildcard covers. Get sloppy enough — a bare repo:* — and any repository on GitHub can assume your role. This is not theoretical; it is a recurring finding in real audits.

Scope it to one repository and one branch. If you need more, add more statements, explicitly.

Decision two: treat Terraform state as a secret, because it is

Terraform state contains every attribute of every resource it manages. Database passwords, generated keys, connection strings. Marking a variable sensitive redacts it from console output and changes nothing about what is written to the state file.

So the state bucket gets four things: server-side encryption with a customer-managed KMS key that rotates, versioning so a corrupted state can be recovered, all four public access block flags, and a bucket policy that refuses plaintext connections outright.

resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

data "aws_iam_policy_document" "deny_insecure_transport" {
  statement {
    sid       = "DenyInsecureTransport"
    effect    = "Deny"
    actions   = ["s3:*"]
    resources = [
      aws_s3_bucket.state.arn,
      "${aws_s3_bucket.state.arn}/*",
    ]

    principals {
      type        = "*"
      identifiers = ["*"]
    }

    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }
}

All four public access block flags, not two. They cover different paths — ACLs and bucket policies, existing objects and future ones — and setting a subset leaves a gap that reads as protected on a dashboard.

There is a chicken-and-egg problem here that catches everyone: the stack that creates the state bucket cannot store its own state in a bucket that does not exist yet. You run it once with local state, then migrate:

cd bootstrap
terraform init
terraform apply
terraform output          # bucket, lock table, KMS key, region

# Now move bootstrap's own state into the bucket it just created
terraform init -migrate-state

Decision three: shut the default security group

Every VPC gets a default security group that permits all traffic between members and all outbound traffic. Anything launched without an explicit security group lands on it. That is a quiet failure mode — nothing errors, the instance works, and it has more network reach than anyone intended.

Terraform can adopt and empty it:

resource "aws_default_security_group" "default" {
  vpc_id = aws_vpc.this.id

  # No ingress and no egress blocks. Terraform strips AWS's rules,
  # so anything that accidentally lands here can reach nothing.
}

Four lines, and misconfiguration becomes loud instead of silent.

Decision four: three subnet tiers, and the bottom one has no way out

The standard two-tier split — public and private — leaves databases in the same tier as application servers. Both can reach the internet through NAT. That is fine right up until something in the data tier is compromised and quietly exfiltrates over 443, which looks like ordinary egress traffic.

• Public — NAT gateways and load balancers. Route to the internet gateway.
• Private — workloads. Outbound through NAT, no inbound from the internet.
• Database — data stores. No default route at all.

The third tier is enforced by absence rather than by a rule:

resource "aws_route_table" "database" {
  vpc_id = aws_vpc.this.id

  # Deliberately no 0.0.0.0/0 route. Not to the IGW, not to NAT.
  # Nothing here can reach the internet, and nothing can reach it.
}

No security group to misconfigure, no NACL to get wrong. The route simply does not exist. That is a much stronger guarantee than a rule someone can widen during an incident and forget to narrow afterwards.

Decision five: gateway endpoints, the one genuinely free win

S3 and DynamoDB gateway endpoints cost nothing. Not “cheap” — zero. They add a route so traffic to those services stays on the AWS backbone rather than going out through NAT.

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.this.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"

  route_table_ids = concat(
    aws_route_table.private[*].id,
    aws_route_table.database[*].id,
  )
}

Two benefits, and the second is the one people miss. Traffic to S3 never traverses NAT, so you stop paying per-gigabyte processing on it. And the isolated database tier can still reach S3 and DynamoDB despite having no internet route, because the endpoint is a route within the VPC rather than a path out of it.

Your Terraform state lives in S3. Without this endpoint, every plan and apply pushes state through the NAT gateway and you are billed for the privilege.

Decision six: SSM Session Manager, not SSH

Port 22 open anywhere means key pairs to distribute, a bastion host to patch, and an audit trail that is whatever auth.log happens to contain.

resource "aws_iam_role_policy_attachment" "ssm_core" {
  role       = aws_iam_role.instance.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

That one attachment gets you shell access with no inbound ports, no key pairs, no bastion. Access is IAM-authorised, so revoking someone is a permissions change rather than a key rotation, and every session can be logged centrally.

Worth knowing the caveat: SSM agents reach the SSM endpoints over the internet by default, so instances in the private tier still need NAT for this. Removing NAT entirely means adding interface endpoints for ssm, ssmmessages and ec2messages, and those cost roughly seven dollars a month per availability zone each.

Decision seven: pin actions to commit SHAs

A Git tag is a movable pointer. If an action’s repository is compromised, an attacker can repoint v4 at whatever they like, and every workflow referencing that tag runs it on the next build — with your OIDC role attached.

steps:
  # A tag can be moved. A commit SHA cannot.
  # Dependabot updates these pins and shows you the diff.
  - uses: actions/checkout@<full-40-character-sha>   # v4.2.2

The comment matters as much as the pin. Without it nobody can tell at a glance which version they are looking at, and the pin becomes something people update blindly.

What it actually costs

Almost all of it is NAT gateways. Everything else is rounding.

• NAT gateways — roughly $32 a month each, plus about $0.045 per gigabyte processed. One in dev, one per availability zone in prod.
• VPC, subnets, route tables, internet gateway — free.
• S3 and DynamoDB gateway endpoints — free.
• KMS keys — about a dollar a month each.
• Flow logs — around $0.50 per gigabyte ingested. Retention is the lever: seven days in dev, a year in prod.
• State bucket and lock table — cents.

Ballpark, that lands around $35 a month for dev and $105 for prod, dominated entirely by NAT. If you only need the isolated tiers and nothing requires outbound internet, dropping NAT takes dev close to free.

Per-AZ NAT in production is an availability decision, not a security one. A single NAT gateway is a single point of failure for outbound traffic across every AZ that routes through it.

Gates that run before any credentials exist

Most of what goes wrong in Terraform is catchable statically. Formatting, syntax, provider misuse, and known-insecure patterns can all be found without touching an AWS account, which means they can run on every commit and in CI without the pipeline holding any cloud credentials at all.

# On every commit, locally
pre-commit install        # gitleaks, fmt, validate, tflint, checkov, terraform-docs

# The same gates CI runs
make all                  # fmt + validate + tflint + checkov

Running the identical command locally and in CI is the part that matters. When those two drift, people stop trusting the local check and start pushing to see what the pipeline says, and the fast feedback loop you built quietly stops existing.

Note also what is missing: there is no terraform plan in that pipeline. Plan needs real credentials, and a static-only pipeline is one that cannot leak anything. A production setup would add a plan job running under the OIDC role and posting to the pull request — but that is a deliberate step up in blast radius, not a default.

What a baseline like this deliberately leaves out

Being clear about scope is more useful than pretending completeness. The reference implementation covers preventative controls and stops there. Missing, on purpose:

• Detective controls. No CloudTrail, Config, GuardDuty or Security Hub. A complete account baseline includes them; this is the preventative half.
• Compute. There is an instance role and profile, but nothing uses them. No ASG, no ECS or EKS, no load balancer.
• Interface endpoints. Only the free gateway ones. Fully private SSM needs paid interface endpoints.
• Multi-region and DR. Versioning covers state recovery. There is no cross-region replication and no failover story.
• Policy-as-code tests. Checkov catches known misconfigurations; there are no Terratest or Conftest assertions on module contracts.

A baseline you can describe the edges of is more useful than one advertised as complete. You know exactly what you still have to build.

Common mistakes

• Using StringLike with a wildcard in an OIDC trust policy. Scope to one repository and one branch.
• Assuming sensitive = true keeps values out of state. It does not.
• Setting two of the four S3 public access block flags and considering the bucket locked down.
• Leaving the default security group as AWS shipped it.
• Putting databases in the same tier as application servers because two tiers felt like enough.
• Skipping gateway endpoints and paying NAT processing charges on your own state file.
• Referencing actions by tag and assuming a tag is immutable.
• Committing backend.hcl or terraform.tfvars. State bucket names embed the account ID.

Frequently asked questions

Is an AWS account baseline the same as a landing zone?

No. A landing zone in the Control Tower sense governs many accounts — organisational units, service control policies, account vending. A baseline is what you lay down inside a single account. Baselines are what you need when you have one or two accounts and Control Tower would be enormous overkill.

Why not just use Control Tower?

Control Tower is a good answer for a multi-account organisation with governance requirements. It is a lot of machinery for a team running a handful of accounts, and it hides the decisions rather than making them legible. Writing the baseline yourself means you can read every choice and change any of them.

Do I really need a separate database subnet tier?

It costs nothing and closes an entire exfiltration path. If your data tier has no route to the internet, a compromised database cannot phone home regardless of what the attacker does with it. Two tiers is not wrong, but the third is close to free.

How do I cut the NAT gateway cost?

Use one NAT for non-production instead of one per AZ. Add gateway endpoints so S3 and DynamoDB traffic bypasses NAT entirely. Beyond that, work out what genuinely needs outbound internet — often less than you assume — and consider dropping NAT for environments that do not.

Can I use this with OpenTofu?

Yes. It is standard HCL against the AWS provider, so it runs on either. Swap terraform for tofu in the commands and everything else is the same.

Where does the OIDC provider itself live?

It is account-global, so only one environment can create it and the others reference it. That is workable but awkward. In a real organisation it belongs in a separate account-level stack that runs before any environment.

Wrapping up

None of the decisions in an AWS account baseline are individually clever. Scope the trust policy. Encrypt the state. Empty the default security group. Give the data tier no way out. Take the free endpoints. Use SSM instead of SSH. Pin your actions. Each one is a few lines of Terraform.

What makes them valuable is doing them before the first workload arrives, because retrofitting network segmentation into a running account is a project, and doing it at the start is an afternoon.

The full implementation is on GitHub at JohnNessime/terraform-aws-baseline, MIT licensed. It is a reference implementation rather than something that has served production traffic — a worked example of how I would lay an account down, with the reasoning written out in the docs. Read it, disagree with parts of it, take what is useful.

Need a baseline laid down properly?

I work with teams on AWS foundations — account structure, network segmentation, IAM that does not rely on long-lived keys, Terraform layouts that survive more than one engineer, and CI pipelines that catch problems before they reach an account.

If you would rather have this done once and done right, you can hire me on Upwork.

The post A Secure-by-Default AWS Account Baseline in Terraform appeared first on John Nessime.

The Pulumi vs Terraform question got harder in the last two years, not easier. The licence changed, the community forked, IBM bought HashiCorp, HashiCorp killed its own general-purpose-language option, and Pulumi started speaking HCL. A lot of the comparison articles you will find were written before any of that happened.

This is an attempt at an honest read of where things actually stand, what the real trade-off is once you strip away the marketing, and how to pick a lane you will not regret in eighteen months.

What actually changed

A short timeline, because the current landscape only makes sense against it.

• August 2023 — HashiCorp relicensed Terraform from MPL 2.0 to the Business Source License. Source-available, free for provisioning your own infrastructure, but you cannot build a competing commercial product on it.
• January 2024 — OpenTofu shipped 1.6.0, a fork of the last MPL-licensed code, governed by the Linux Foundation.
• February 2025 — IBM completed its acquisition of HashiCorp, valued at around $6.4 billion.
• December 2025 — HashiCorp deprecated CDK for Terraform and archived the repository. The official way to write Terraform in a general-purpose language no longer exists.
• Also December 2025 — Pulumi announced native HCL support and the ability to work with Terraform and OpenTofu state.

Those last two matter more than the licence drama, and they point in opposite directions. HashiCorp closed the door on writing Terraform in a real language. Pulumi opened the door to writing Pulumi in HCL. Whatever else you conclude, “HCL versus real languages” is no longer a clean line between two products.

The real axis: a configuration language or a programming language

Strip out the licensing and the corporate ownership and one genuine difference remains. HCL is a purpose-built configuration language that describes desired state. Pulumi runs an actual program that constructs a resource graph, and then applies it.

Here is the same trivial thing in both. Terraform:

resource "aws_s3_bucket" "logs" {
  for_each = toset(var.environments)
  bucket   = "acme-logs-${each.key}"

  tags = {
    Environment = each.key
    ManagedBy   = "terraform"
  }
}

And Pulumi, in TypeScript:

import * as aws from "@pulumi/aws";

const environments = ["dev", "staging", "prod"];

const buckets = environments.map(env =>
  new aws.s3.Bucket(`logs-${env}`, {
    bucket: `acme-logs-${env}`,
    tags: {
      Environment: env,
      ManagedBy: "pulumi",
    },
  })
);

For something this small the difference is aesthetic. It stops being aesthetic around the third nested conditional.

What you actually gain with a general-purpose language

• Real abstractions. Classes, functions, inheritance. A Pulumi component that encapsulates “our standard service” is just a class, and it composes the way code composes.
• Real testing. Jest, pytest, Go’s testing package. You can unit test infrastructure logic with the same tools and the same CI as the rest of your codebase.
• The package ecosystem. Need to parse a CSV of subnets, call an internal API, or do something genuinely awkward? It is an npm install rather than an external data source and a shell script.
• IDE support that works. Type checking, refactoring, jump-to-definition. HCL tooling has improved a lot but it is not the same thing.

What you lose, which comparison posts skip

This is the part worth reading twice, because it is where the regret comes from.

• Constraint is a feature. HCL is limited on purpose. When a language cannot express clever things, nobody writes clever things. Give a team a real language and someone will eventually build a metaprogramming layer that generates resource names from a database lookup, and reviewing that pull request will be nobody’s idea of a good afternoon.
• Readability across skill levels. A network engineer who has never written TypeScript can usually read HCL and tell you what it provisions. The reverse is not true. If your infrastructure is touched by people who are not primarily programmers, that matters enormously.
• The hiring pool. There are far more engineers with Terraform on their CV than Pulumi. That is not an argument about quality, it is an argument about how long it takes to backfill a role.
• A dependency on a runtime. Pulumi programs need Node, Python, or Go present and correct in every environment that runs them, including CI. It is one more thing that can drift.
• Asynchrony leaks into the model. Resource outputs are not plain values. They are futures, and you have to handle them as such. Every Pulumi team hits this on day two.

# This does not do what a newcomer expects. bucket.id is an Output,
# not a string, so you get an object reference in the name.
# bucket_name = f"logs-for-{bucket.id}"

# Correct: work inside apply, where the value is resolved.
bucket_name = bucket.id.apply(lambda bid: f"logs-for-{bid}")

None of these are dealbreakers. They are just the bill that arrives later, and it is worth reading before you sign.

State, providers, and the things that actually break

State management

Both tools keep a state file mapping your code to real resources, and both will ruin your afternoon if that file gets corrupted or out of sync.

Terraform and OpenTofu use pluggable backends: S3 with DynamoDB locking, Azure Blob, GCS, Consul, or HCP Terraform. OpenTofu added client-side state encryption, which Terraform does not have natively and which is a genuine advantage if your compliance people care about what is sitting in that bucket.

Pulumi defaults to Pulumi Cloud, which is free for individuals and paid for teams. It also supports self-managed backends, and you should know this because the default sometimes gets mistaken for a requirement:

# Self-managed state in S3, no Pulumi Cloud account involved
pulumi login s3://my-pulumi-state-bucket

# Or entirely local, useful for experiments
pulumi login --local

Pulumi encrypts secrets in state by default. With Terraform, values marked sensitive are redacted in output but still sit in plaintext in the state file, which surprises people who assumed otherwise.

The provider ecosystem, and a myth worth killing

You will read that Pulumi has a much smaller provider ecosystem. That was true once and is mostly no longer the point, because Pulumi bridges Terraform providers. If a Terraform provider exists, you can almost certainly use it from Pulumi, either pre-built in the Pulumi Registry or generated on demand.

Two caveats that are real:

• Bridged providers can lag upstream releases. If you need a resource type that shipped last week, check before committing.
• When something breaks deep in a bridged provider, you are debugging across a translation layer. That is a worse afternoon than debugging the provider directly.

Pulumi also ships native providers for Kubernetes and Azure, generated from the platform APIs, which give same-day coverage of new resources rather than waiting on a bridge.

Policy and governance

Terraform’s first-class answer is Sentinel, which is tied to HCP Terraform. OpenTofu users generally reach for OPA or Conftest. Pulumi has CrossGuard, where policies are written in the same languages as everything else.

If you are in a regulated environment and your auditors already know what Sentinel is, that is a heavier thumb on the scale than any language preference.

Licensing, since it is what forced the question

• Terraform — BSL 1.1. Source-available. Fine for provisioning your own infrastructure. Not fine for building a product that competes with HashiCorp. Each release converts to an open licence four years after publication, but only that release.
• OpenTofu — MPL 2.0, OSI-approved, Linux Foundation governance. No single vendor can relicense it.
• Pulumi — Apache 2.0 for the engine and CLI. Permissive, with a patent grant. Pulumi Corp monetises through Pulumi Cloud rather than through the licence.

For the overwhelming majority of teams, the BSL changes nothing about daily work. You are provisioning your own infrastructure and that is explicitly permitted. The concern is not today’s terms, it is that a single vendor demonstrated it can change them, and that vendor is now part of IBM.

If your objection to Terraform is licensing rather than language, the answer is OpenTofu, not Pulumi. Switching paradigms to solve a legal problem is an expensive way to do it.

Choosing a lane

Three coherent positions. Pick the one that matches your constraints rather than the one that sounds most modern.

Lane one: OpenTofu

The default for most teams already running Terraform who are uneasy about the licence. Migration is genuinely close to free: same HCL, same providers, same state format, same commands.

# Back up state first. Always.
terraform state pull > backup-$(date +%F).tfstate

# Then initialise with tofu against the same backend
tofu init
tofu plan

A clean plan with no proposed changes means you are done. That is the whole migration for most codebases.

Choose this if: you have existing Terraform, HCL is working for you, and you want open governance without retraining anyone.

Lane two: Terraform with HCP

Still the biggest ecosystem, the most documentation, the most Stack Overflow answers, the most people who already know it. Stacks and ephemeral resources are Terraform-only. Sentinel is Terraform-only. If you are deep in HCP Terraform and it works, the licence is an abstract concern rather than a practical one.

Choose this if: you are in a regulated industry, you need vendor support with a contract behind it, or your organisation is standardising on IBM and Red Hat tooling anyway.

Lane three: Pulumi

The strongest case for Pulumi is not “real languages are nicer”. It is when your infrastructure genuinely needs logic that HCL fights you on: dynamic topologies, infrastructure derived from application code, platform teams building abstractions for other teams to consume. If you are writing a self-service platform rather than a set of environments, a real language stops being a preference and starts being the right tool.

Choose this if: your team is primarily software engineers, your infrastructure has real branching logic, or you are building a platform product rather than provisioning a fixed estate.

Do not choose it because writing loops in HCL annoyed you once.

What migrating actually looks like

Terraform to OpenTofu is a swap. Terraform to Pulumi is a refactor, and anyone telling you otherwise is selling something.

Pulumi does provide real tooling for it:

# Convert HCL into a Pulumi program in your chosen language
pulumi convert --from terraform --language typescript

# Adopt the resources described by an existing state file
pulumi import --from terraform ./terraform.tfstate

Pulumi can also consume Terraform modules directly, so existing module investment is not automatically wasted. That is the single most useful fact for anyone considering a gradual move.

A sane sequence:

1. Pick one small, low-risk stack. Not the production VPC.
2. Convert it, import the state, and run a preview. The only acceptable result is no changes.
3. Refactor the generated code. Converted output is mechanically correct and stylistically awful, which is fine, but do not leave it that way.
4. Run both tools side by side for a while. Coexistence is supported; a big-bang cutover is not necessary.
5. Only then decide whether to move anything else.

If step two produces a diff you cannot explain, stop. An unexplained diff during an IaC migration is how resources get destroyed and recreated.

Common mistakes

• Switching tools to solve a licensing concern when OpenTofu solves it for free.
• Picking the tool the loudest engineer prefers rather than the one the whole team can maintain.
• Adopting CDKTF in 2026. It was archived in December 2025 and will not receive fixes.
• Treating a converted program as finished code rather than a starting point.
• Assuming Terraform state is safe because values are marked sensitive. Encrypt the backend.
• Migrating production first because it is the stack that matters most. It is the stack that matters most.
• Running two tools against the same resources without a clear ownership boundary.

Best practices whichever lane you pick

• Remote state with locking and encryption, from the first commit. Never local state for anything shared.
• Small stacks with clear boundaries. One enormous state file is the most common self-inflicted wound in this field.
• Plan or preview output in every pull request, and require a human to read it.
• Pin provider versions. Floating versions turn an unrelated deploy into an incident.
• Policy as code before you need it, not after the audit.
• Document the decision. Six months from now nobody will remember why, and the reasoning is worth more than the choice.

Frequently asked questions

Is Pulumi better than Terraform?

Neither is better in the abstract. Pulumi suits teams of software engineers building infrastructure with real logic in it. Terraform and OpenTofu suit teams provisioning a defined estate where constraint and readability matter more than expressiveness.

Can I use Terraform providers with Pulumi?

Yes. Pulumi bridges Terraform providers, either pre-built in its registry or generated on demand, and can consume Terraform modules directly. Bridged providers can lag upstream releases, so verify coverage for anything recent.

Should I move from Terraform to OpenTofu?

If licensing is your concern, it is close to free: same HCL, same providers, same state. Back up state, run tofu init and tofu plan, and confirm no changes. If you rely on Stacks, Sentinel, or HCP Terraform, staying put is reasonable.

What happened to CDKTF?

HashiCorp deprecated it on 10 December 2025 and archived the repository. It gets no further fixes or compatibility updates. If you want general-purpose languages for infrastructure now, Pulumi is the maintained option.

Does the BSL licence stop me using Terraform commercially?

Not for provisioning your own infrastructure, which is what almost everyone does. It restricts building a competing commercial IaC product on top of it. Check with your own legal team rather than a blog post if you are anywhere near that line.

Can Pulumi and Terraform coexist?

Yes, and for any migration of size that is the sensible approach. Pulumi can reference Terraform state and consume Terraform modules. The rule is that each resource has exactly one owner.

Wrapping up

The Pulumi vs Terraform decision is less about the tools than about who maintains your infrastructure and what it has to do. If your estate is a known set of environments maintained by people with mixed backgrounds, a constrained configuration language is a feature and OpenTofu gives you that without licensing anxiety. If you are building a platform, with branching logic and abstractions other teams consume, a real language earns its complexity and Pulumi is the maintained way to get one.

What has genuinely changed is that the lanes are no longer walled off. Pulumi speaks HCL and reads Terraform state. OpenTofu is a drop-in for Terraform. Converting between them is tooling rather than a rewrite. That makes the decision less permanent than it feels, which is a good reason to pick deliberately and a bad reason to agonise.

Pick the lane your team can still maintain when the person who chose it has moved on. Then write down why.

Need help choosing or migrating?

I work with teams on infrastructure as code decisions and the migrations that follow — Terraform to OpenTofu, Terraform to Pulumi, splitting up state files that grew too large, and wiring the whole thing into CI with policy checks that actually run.

If you would rather talk it through with someone who has untangled a few of these, you can hire me on Upwork.

The post Pulumi vs Terraform: Choosing a Lane appeared first on John Nessime.

docker compose up -d

A web application, database, cache, monitoring service, and reverse proxy can all be launched in seconds. Everything is isolated, repeatable, and relatively easy to move between environments.

Then, eventually, something breaks.

A container refuses to start. A port is already occupied. A database loses access to its storage directory. One service cannot resolve the hostname of another. A reverse proxy returns a 502 error even though the application appears to be running.

That is usually the point where Docker stops feeling like a convenient deployment tool and starts becoming a real infrastructure skill.

This guide is based on the way I approach Docker issues on real Linux servers. It is not a list of commands to copy blindly. It is a troubleshooting method for discovering what failed, understanding why it failed, and solving it without making the situation worse.

Docker Is Usually Not the Real Problem

When a container fails, Docker is often blamed first. In many cases, however, Docker is only exposing a problem somewhere else in the system.

• Incorrect Linux file permissions
• A missing environment variable
• An unavailable database
• A port conflict on the host
• An incorrect volume mount
• A DNS or network configuration problem
• A reverse proxy pointing to the wrong destination
• Insufficient memory or disk space
• An application-level configuration error

The container runtime may be working exactly as designed. The application inside the container may simply be unable to operate under the conditions it was given.

This distinction matters because repeatedly deleting and rebuilding containers rarely fixes the underlying issue. It may temporarily hide it, but the same failure usually returns.

My First Rule of Docker Troubleshooting: Do Not Delete Anything Yet

When a container suddenly exits, the first instinct is often to remove it, rebuild the image, or restart the entire stack. That destroys useful evidence.

Before changing anything, I start by collecting information.

docker ps -a

This shows running and stopped containers, their current states, exposed ports, names, and recent exit information.

docker logs container-name

The logs frequently reveal the immediate cause: a missing file, invalid password, failed database connection, permission error, or malformed configuration.

docker inspect container-name

Inspection provides the container’s network settings, environment variables, volume mounts, restart policy, health status, image information, and runtime configuration.

These three commands answer a large percentage of Docker troubleshooting questions before any corrective action is required.

Understanding Docker Container States

A container can exist in several states, and each state tells a different story.

Running

The main process inside the container is active. This does not automatically mean the application is healthy or ready to receive traffic.

Exited

The container’s main process stopped. It may have completed normally, crashed, or failed during startup. The exit code and logs provide the next clues.

Restarting

The container repeatedly starts and fails while Docker applies its restart policy. This commonly indicates an application crash, invalid configuration, inaccessible dependency, or permission problem.

Created

The container was created but its main process has not started successfully.

Dead

Docker could not properly stop or remove the container. This is less common and may indicate a lower-level runtime, storage, or host issue.

Knowing the state prevents random troubleshooting. A networking issue requires a different investigation from a process that exits because of a syntax error.

Start With the Container Logs

Logs are normally the fastest route to the cause of a Docker issue.

docker logs --tail 100 container-name

This displays the latest 100 lines without flooding the terminal with the container’s entire history.

docker logs -f container-name

The follow option streams new log entries in real time. It is useful while restarting a service or reproducing a problem.

docker logs --since 30m container-name

This limits the output to a recent period, which is especially useful for long-running containers.

When reading logs, I do not look only at the final line. The true cause often appears several lines earlier. The last error may simply be a consequence of the first failure.

Docker Networking: Where Many Problems Begin

Docker networking is one of the most common sources of confusion, especially when several containers need to communicate.

Imagine a stack containing:

• Grafana
• Prometheus
• A web application
• MySQL or PostgreSQL
• Redis
• Nginx or Apache

Each service may work independently while communication between them fails.

The Localhost Mistake

Inside a container, localhost refers to that container itself. It does not refer to the Docker host, and it does not refer to another container.

For example, if Grafana needs to connect to Prometheus, the data source address should usually use the Docker Compose service name:

http://prometheus:9090

Using the following from inside the Grafana container would normally point back to Grafana itself:

http://localhost:9090

This small misunderstanding causes a surprising number of connection failures.

Confirm That Containers Share a Network

docker network ls

docker network inspect network-name

The inspection output shows which containers are attached to a network and which IP addresses were assigned.

Containers on unrelated networks cannot automatically communicate by service name. They must share a Docker network or use another explicitly configured route.

Test Connectivity From Inside the Container

docker exec -it container-name sh

Depending on the image, Bash may be available instead:

docker exec -it container-name bash

From inside the container, test name resolution and connectivity using the utilities available in that image:

getent hosts database
curl http://application:8080
wget -qO- http://prometheus:9090/-/ready

A successful result confirms that Docker DNS and network routing are functioning. A failed result narrows the investigation considerably.

Host Ports and Container Ports Are Different

A Docker Compose port mapping such as the following is read from left to right:

ports:
  - "8080:80"

Port 8080 belongs to the Docker host. Port 80 belongs to the application inside the container.

Traffic sent to the host on port 8080 is forwarded to port 80 inside the container.

If another service already occupies host port 8080, Docker cannot bind to it.

ss -tulpn

To check a specific port:

ss -tulpn | grep :8080

A common mistake is changing the container’s internal port when only the host-side mapping needs to change.

For example, this avoids a conflict while leaving the application untouched:

ports:
  - "8081:80"

Published Ports Are Not Required for Container-to-Container Traffic

Containers connected to the same Docker network can usually communicate through their internal ports without publishing those ports to the host.

For example, Prometheus can communicate with an exporter on the shared Docker network without exposing that exporter publicly.

This reduces unnecessary exposure and produces a cleaner architecture. Publish only the ports that must be reached from the host, a reverse proxy, or an external client.

Docker Volumes: Persistent Storage Done Properly

Containers should be treated as disposable. Persistent data should not depend on the writable filesystem inside a container.

Without a correctly configured volume, deleting a database container may also delete the database stored inside it.

Named Volumes

services:
  database:
    image: mariadb:latest
    volumes:
      - database-data:/var/lib/mysql

volumes:
  database-data:

Docker manages the storage location while the application writes to its expected internal directory.

Bind Mounts

volumes:
  - ./config:/etc/application

A bind mount maps a specific host path into the container. It is useful for configuration files, development directories, and data that administrators need to access directly.

Bind mounts also introduce more responsibility. The host directory must exist with the correct permissions, ownership, security context, and contents.

Inspect Existing Volumes

docker volume ls

docker volume inspect volume-name

Before deleting an apparently unused volume, confirm which application created it and whether it contains irreplaceable data.

Permission Problems Are Quiet but Destructive

Permission errors can prevent databases from initializing, web applications from uploading files, monitoring tools from writing data, or services from reading configuration files.

The first checks are simple:

ls -lah
stat path-to-file
stat path-to-directory

The user inside the container may have a different numeric user ID from the owner of the host directory. The names do not matter as much as the numeric UID and GID.

docker exec container-name id

Compare that result with the ownership of the mounted host path.

Changing permissions to 777 may appear to solve the issue, but it is usually an unsafe shortcut rather than a proper solution. Correct ownership and the minimum required permissions are preferable.

SELinux Can Affect Docker Bind Mounts

On distributions such as AlmaLinux, Rocky Linux, CentOS Stream, Fedora, and Red Hat Enterprise Linux, standard Unix permissions may look correct while SELinux still blocks access.

Docker Compose bind mounts may require an SELinux relabeling option:

volumes:
  - ./data:/var/lib/application:Z

The uppercase Z assigns a private SELinux label for the container. A lowercase z is used when the content must be shared between multiple containers.

SELinux should not be disabled simply to make a container work. The better approach is to identify the denied access and apply the correct labeling or policy.

Environment Variables Can Break an Entire Stack

Modern containerized applications depend heavily on environment variables for database credentials, API keys, application URLs, encryption secrets, feature flags, and runtime settings.

One missing or incorrectly escaped value can stop a service from starting.

docker compose config

This command renders the resolved Docker Compose configuration. It helps reveal missing substitutions, merged settings, malformed values, and unexpected defaults.

To inspect the environment available inside a running container:

docker exec container-name env

Be careful when sharing this output because it may include passwords, tokens, and private service credentials.

Docker Compose Configuration Errors

YAML is readable, but it is sensitive to indentation and structure. A service can be valid YAML while still being configured incorrectly for Docker Compose.

Before deploying a change, validate the file:

docker compose config --quiet

If the command returns without an error, the configuration is structurally valid.

For a complete rendered version:

docker compose config

This is especially valuable when Compose files use environment substitutions, overrides, extension fields, or multiple configuration files.

Health Checks Reveal More Than Running Status

A container can be running while the application inside it is unusable.

A database may still be initializing. A web service may be listening but unable to connect to its database. An API may be running but returning errors for every request.

A health check lets Docker test actual application readiness.

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 20s

The application should expose a lightweight endpoint that confirms its critical dependencies are available.

To inspect the current status:

docker inspect --format='{{json .State.Health}}' container-name

A health check does not repair an unhealthy service by itself, but it provides a far more accurate operational signal than process status alone.

Restart Policies Can Hide Repeated Failures

Restart policies are useful in production, but they can create the impression that a service is stable when it is actually crashing continuously.

restart: unless-stopped

If the application exits every few seconds, Docker restarts it every few seconds. The container may appear in the process list, but the service may never become ready.

Check the restart count:

docker inspect --format='{{.RestartCount}}' container-name

A rapidly increasing number indicates that the restart policy is masking an unresolved failure.

Reverse Proxy Errors: 502, 504, SSL Problems, and Redirect Loops

Reverse proxies such as Nginx, Apache, Traefik, and Caddy often sit between users and containerized applications.

When the public website fails, the application container may still be perfectly healthy. The proxy may simply be unable to reach it.

Common Reverse Proxy Problems

• The upstream hostname is incorrect.
• The proxy uses the host port when it should use the container port.
• The proxy and application do not share a Docker network.
• The application listens only on 127.0.0.1 inside the container.
• The proxy forwards HTTP while the backend expects HTTPS.
• The application is unaware that the original request used HTTPS.
• The public hostname does not match the certificate.
• Both the proxy and application force redirects, creating a loop.

Test the application directly before blaming the proxy:

curl -I http://127.0.0.1:published-port

If the application responds directly but fails through the domain, the reverse proxy, firewall, DNS, or TLS configuration becomes the main area of investigation.

An Application Must Listen on the Correct Interface

An application inside a container may start successfully while listening only on 127.0.0.1. In that situation, Docker’s network interface may not be able to reach it.

Containerized web applications should usually listen on:

0.0.0.0

This allows the process to accept connections through the container’s network interfaces.

This setting is commonly controlled by application arguments or environment variables such as HOST, BIND_ADDRESS, or LISTEN_ADDRESS.

Resource Exhaustion Can Look Like a Docker Failure

Containers share the CPU, memory, storage, and network resources of the host. One service can consume enough resources to affect every other workload.

docker stats

This provides a live view of CPU, memory, network, and block I/O usage for running containers.

On the host, I also check:

free -h
df -h
df -i
uptime

Disk space is not the only storage limit. A filesystem can also run out of inodes, preventing new files from being created even when gigabytes remain available.

Kernel logs may reveal out-of-memory termination:

dmesg | grep -i -E "out of memory|killed process|oom"

If the kernel killed the process, the application logs may end suddenly without a useful explanation.

Docker Disk Usage Grows Quietly

Old images, stopped containers, build cache, unused volumes, and unbounded logs can gradually consume the host’s storage.

docker system df

For more detail:

docker system df -v

Cleanup commands should be used carefully:

docker image prune
docker container prune
docker builder prune

A broad command such as docker system prune can remove more than expected. Never add the volume-removal option unless the contents of unused volumes have been verified and backed up.

Container Logs Can Fill the Server

Docker’s default JSON logging driver can create large log files when applications produce continuous output.

Log rotation can be configured in Docker Compose:

logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "5"

This limits the number and size of local log files for that container.

For production systems, centralized logging with tools such as Grafana Loki, Elasticsearch, OpenSearch, or a managed logging service makes incident investigation considerably easier.

Image Architecture and Compatibility Problems

An image built for one CPU architecture may not run correctly on another. This becomes relevant when moving between AMD64 servers, ARM-based cloud instances, Apple Silicon machines, and single-board computers.

Check the host architecture:

uname -m

Inspect the image:

docker image inspect image-name

Errors mentioning an invalid executable format frequently indicate an architecture mismatch.

Multi-platform builds can be produced using Docker Buildx:

docker buildx build --platform linux/amd64,linux/arm64 .

A New Image Tag Does Not Always Mean a Safe Upgrade

Using latest may be convenient, but it makes deployments less predictable. A future image update can introduce a breaking configuration change, database migration, removed feature, or incompatible dependency.

Production deployments are usually safer with an explicit version:

image: grafana/grafana:12.0.2

Before upgrading:

• Read the release notes.
• Confirm supported upgrade paths.
• Back up persistent data.
• Record the current working version.
• Test the change outside production where possible.
• Prepare a rollback procedure.

Containers make software easier to replace, but they do not make every application upgrade reversible.

Dependency Startup Order Is Not Readiness

Docker Compose can start services in a defined order, but starting a database container does not mean the database is immediately ready for connections.

An application may launch too early, fail to connect, and exit.

Health checks and dependency conditions can improve this:

services:
  database:
    image: postgres:17
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  application:
    image: example/application:1.0
    depends_on:
      database:
        condition: service_healthy

The application itself should still handle temporary dependency failures gracefully. Networks fluctuate, databases restart, and external services occasionally become unavailable.

Backups Matter More Than Recreating Containers

Application containers are normally easy to recreate. Persistent business data is not.

A reliable Docker backup strategy should cover:

• Database-native backups
• Named volumes and bind-mounted data
• Docker Compose files
• Environment files and secrets
• Reverse proxy configurations
• TLS and certificate configuration
• Application-specific settings
• A tested restoration procedure

A backup that has never been restored is only an assumption.

Database backups should normally use the database’s own tools rather than copying active database files blindly. Examples include pg_dump for PostgreSQL and mysqldump or mariadb-dump for MySQL-compatible systems.

Monitoring Changes Troubleshooting From Guesswork to Evidence

Without monitoring, administrators usually investigate after users report a failure. By then, the container may have restarted and the original conditions may no longer be visible.

Metrics help answer better questions:

• When did memory consumption begin increasing?
• Which container experienced the first error?
• Was the disk already nearly full?
• Did network latency rise before the application failed?
• How many times did the service restart?
• Did the host run out of memory?
• Was the reverse proxy still able to reach the backend?

Prometheus, Grafana, node-level exporters, container metrics, uptime probes, and centralized logs provide the historical context that individual Docker commands cannot.

A Practical Docker Troubleshooting Workflow

When a Docker service fails, I use a sequence rather than jumping between unrelated commands.

1. Identify the Exact Symptom

• Does the container fail to start?
• Does it start and then exit?
• Is it running but unreachable?
• Is the application responding with errors?
• Is the public domain failing while the local service works?
• Is the problem intermittent?

2. Check Container Status

docker ps -a

3. Read the Logs

docker logs --tail 200 container-name

4. Inspect Runtime Configuration

docker inspect container-name

5. Validate Docker Compose

docker compose config

6. Check Host Resources

docker stats
free -h
df -h
df -i

7. Test Internal Connectivity

Enter the relevant container and test DNS resolution, ports, and HTTP endpoints from the same network context as the application.

8. Check Volumes and Permissions

Confirm that mounts point to the expected locations and that the container user can read or write them as required.

9. Isolate the Reverse Proxy

Test the backend directly. If the backend works, investigate the proxy, DNS, TLS, firewall, and forwarded headers.

10. Change One Thing at a Time

Multiple simultaneous changes make it difficult to identify the actual solution. Apply one correction, test it, and record the result.

Commands I Frequently Use During Docker Investigations

# Show running containers
docker ps

# Show all containers
docker ps -a

# Read recent logs
docker logs --tail 100 container-name

# Follow logs
docker logs -f container-name

# Inspect a container
docker inspect container-name

# Enter a container
docker exec -it container-name sh

# View live resource use
docker stats

# List networks
docker network ls

# Inspect a network
docker network inspect network-name

# List volumes
docker volume ls

# Inspect a volume
docker volume inspect volume-name

# Validate Compose configuration
docker compose config

# View Compose service status
docker compose ps

# Restart one service
docker compose restart service-name

# Recreate one service
docker compose up -d --force-recreate service-name

# View Docker disk usage
docker system df

# View host listening ports
ss -tulpn

# Check disk usage
df -h

# Check inode usage
df -i

# Check memory
free -h

Docker Troubleshooting Mistakes to Avoid

Deleting Containers Before Reading Their Logs

This removes evidence and often leaves the original problem unresolved.

Using Docker System Prune Without Reviewing the Impact

Cleanup commands can remove useful images, build cache, stopped containers, and potentially important storage.

Using Latest Everywhere

Uncontrolled image updates reduce reproducibility and can introduce breaking changes unexpectedly.

Opening Every Port Publicly

Internal services should remain internal unless external access is genuinely required.

Setting Permissions to 777

This weakens security and avoids fixing the actual ownership or access-control issue.

Restarting the Entire Server Too Early

A restart may temporarily restore service while destroying evidence about the original failure.

Changing Several Variables at Once

Even when the issue disappears, it becomes impossible to know which change solved it.

Docker Compose Is Also Infrastructure Documentation

A well-organized Compose file documents the structure of an application environment.

• Which services exist
• Which image versions are deployed
• Which networks connect them
• Which ports are externally accessible
• Where persistent data is stored
• Which environment variables are required
• Which services depend on others
• Which health checks are available
• How containers restart after failure

Six months after a deployment, this information is often more useful than a separate document that was never updated.

Readable service names, comments for non-obvious decisions, pinned image versions, organized environment files, and clear volume names make future troubleshooting much faster.

What Docker Failures Actually Teach

Some of the most useful infrastructure lessons come from deployments that did not work as expected.

A failed container may teach you how Linux permissions interact with numeric user IDs. A 502 error may reveal the difference between host ports and container ports. A disappearing database may demonstrate why persistent volumes matter. An unreachable service may expose a misunderstanding about Docker DNS or network boundaries.

These incidents are frustrating, but they build practical knowledge that clean tutorial environments rarely provide.

Production systems do not normally fail in textbook ways. Several small issues may combine: a nearly full disk, oversized logs, an automatic image upgrade, and an aggressive restart policy can create one confusing incident.

Final Thoughts

Docker makes application deployment more portable, repeatable, and manageable, but it does not remove the need to understand the systems underneath it.

Effective Docker troubleshooting requires knowledge of Linux, networking, storage, process management, application configuration, security, and monitoring.

The most important habit is simple: follow the evidence.

Check the container state. Read the logs. Inspect the configuration. Test the network from the correct location. Verify volumes and permissions. Check the host’s resources. Separate the application from the proxy. Change one thing at a time.

The next time a Docker container fails, resist the urge to delete everything and begin again. The answer is usually already present in the logs, runtime configuration, network design, storage mapping, or host environment.

Docker troubleshooting becomes much less intimidating once every failure is treated as an investigation rather than an emergency.

Frequently Asked Questions About Docker Issues

Why does my Docker container exit immediately?

A Docker container exits when its main process stops. Common causes include invalid configuration, missing environment variables, permission errors, failed dependency connections, and incorrect startup commands. Use docker logs and docker inspect to identify the cause.

Why is my Docker container running but not accessible?

The application may be listening on the wrong interface, the port may not be published correctly, a firewall may be blocking traffic, or the reverse proxy may point to the wrong hostname or port. Test the application both inside the container and directly through the host port.

Why can one Docker container not connect to another?

The containers may not share a Docker network, the wrong service name may be used, or the destination service may not be listening on the expected port. Inside Docker, use the Compose service name rather than localhost.

How do I find which process is using a Docker port?

Run ss -tulpn on the Docker host and filter for the relevant port. If the host port is already occupied, change the host side of the Docker port mapping or stop the conflicting service.

Can restarting Docker solve container problems?

Restarting Docker may temporarily restore a service, but it can also hide the original cause. Logs, container state, disk space, memory, network connectivity, and volume permissions should be checked before restarting the Docker daemon or server.

How can I prevent Docker logs from filling the disk?

Configure log rotation using the Docker logging options, such as max-size and max-file. Production environments may also benefit from centralized logging with Grafana Loki, OpenSearch, Elasticsearch, or a managed logging platform.

Should I use the latest Docker image tag?

The latest tag is convenient but unpredictable. Pinning a tested image version improves repeatability and makes upgrades and rollbacks easier to control.

Are Docker volumes automatically backed up?

No. Docker volumes provide persistent storage, but Docker does not automatically create external backups. Important volumes and databases need a separate, tested backup and restoration process.

The post Docker Issues & Troubleshooting: Real-World Solutions appeared first on John Nessime.