You go back through the logs and there it is, four hundred lines up, in a step that reported success: a download that returned an HTML error page, got piped into tar, and produced an archive containing nothing useful. The shell reported the exit status of tar, not of curl. The build carried on. The pipeline was never lying on purpose. It just never asked the right question.

Two things go wrong with pipelines, and only one of them gets talked about. The famous one is slow feedback: thirty-four minutes of build and integration tests before a linter tells you a file is missing a trailing newline. The expensive one is a pipeline that passes when it should have stopped. A fail fast pipeline fixes the first problem. Making it explain itself fixes the second, and that is the half most teams skip.

This post covers both: how to order stages so real failures surface in seconds, how to stop the shell from hiding non-zero exits, how to design failure output that hands the next engineer the fix, and the troubleshooting steps for when a pipeline stubbornly refuses to fail at the right moment.

The failure that actually bites: a pipeline that lies

Start here, because everything else is optimisation on top. A slow pipeline wastes your afternoon. A silently passing pipeline ships a broken artifact and burns a day of debugging in production, usually while somebody is on a call with a customer.

The shell is where most of this happens. Bash gives you three defaults that are wrong for CI, and one line fixes all three.

#!/usr/bin/env bash
set -euo pipefail

# Without pipefail this line exits 0 even when curl fails,
# because the shell reports the status of the LAST command
# in the pipe, which is tar.
curl -sSL https://example.com/artifact.tar.gz | tar -xz

What each flag actually protects

• -e stops the script when a command exits non-zero. Without it, a failed database migration is followed by a successful echo, and the script exits 0 because the last command succeeded.
• -u turns an unset variable into an error. This is the one that saves you from rm -rf "$PREFIX/build" expanding to rm -rf "/build" when someone renames an environment variable in the pipeline settings and forgets the script.
• -o pipefail makes a pipeline return the first non-zero status instead of the last one. Every curl | tar, cat | jq, and docker logs | grep in your scripts is currently reporting only the exit code of the thing on the right.

The exit status is only half the story though. Plenty of tools return 0 while doing something useless, and curl is the classic offender.

# curl exits 0 here even when the server returns 500,
# and you end up untarring an HTML error page.
curl -sSL "$ARTIFACT_URL" | tar -xz

# --fail          makes curl exit non-zero on 4xx and 5xx
# --show-error    keeps the reason visible even with --silent
# --location      follows redirects instead of saving a 302 body
curl --fail --silent --show-error --location "$ARTIFACT_URL" | tar -xz

Now the download failing stops the build, and pipefail makes sure that failure propagates past the pipe. The first fixes the check, the second makes sure the check is heard.

Where set -e quietly gives up

This trips people up constantly. set -e is suspended whenever a command’s exit status is being consumed by something else. It is not a safety net, it is a default.

set -e

# -e does NOT stop the script here.
# The if statement consumes the exit status.
if grep -q "TODO" src/main.go; then
  echo "found"
fi

# -e does NOT stop the script here either.
# The left side of || is explicitly allowed to fail.
run_migrations || echo "migrations failed, continuing anyway"

# -e DOES stop the script here.
run_migrations

The same trap applies inside containers. On Debian-based images /bin/sh is dash, which has no pipefail option at all, so putting set -o pipefail inside a RUN line fails outright. Set the shell at the top of the Dockerfile instead. Hadolint flags this as DL4006 and it is worth wiring into your static stage.

# Set this before any RUN instruction that contains a pipe.
SHELL ["/bin/bash", "-o", "pipefail", "-c"]

RUN curl --fail --silent --show-error --location 
      https://example.com/tool.tar.gz | tar -xz -C /usr/local/bin

Order stages by cost, not by category

Most pipelines are laid out the way the team thinks about work: tests, then build, then security, then deploy. That grouping is comfortable and it is the wrong axis. Order by two things instead, cost and failure probability.

The rule I use: every stage should be cheaper than the one after it and more likely to fail than the one after it. If a check catches ten percent of your broken commits and takes fifteen seconds, it goes in front of the check that catches two percent and takes nine minutes. That is the whole design principle behind a fail fast pipeline.

1. Syntax and static analysis. shellcheck, hadolint, yamllint, terraform validate, tflint, formatter checks, type checks. Seconds, no dependencies, no network. This stage catches an embarrassing share of broken pushes.
2. Unit tests. No database, no containers, no network. If they need any of those, they are integration tests and belong further down.
3. Secrets and dependency scanning. Fast, and a leaked credential is worth stopping before you build an image around it.
4. Build. This is where you start spending real minutes and cache space.
5. Integration and end-to-end. Slow, flaky, expensive to run. Everything above should have already caught the boring failures.
6. Deploy and post-deploy verification.

A note on the trade-off, because this ordering costs something. Splitting the fast checks into their own stage adds a checkout and a container pull before your unit tests even start. On a busy shared runner that overhead is real. If your static stage takes twelve seconds of work and forty seconds of setup, merge it into the test job and run it as the first step instead. The principle is failure order, not job count.

The other thing that keeps the fast lane fast is caching, and it is the piece people get wrong in the opposite direction: they cache so aggressively that a stale dependency tree hides a genuine break. Cache the package manager’s download directory, not the resolved dependency tree, and key the cache on the lockfile hash.

Let the platform kill work you already know is doomed

Ordering stages is only useful if the platform actually stops when something breaks. Three settings do most of the work, and they exist under different names on every runner.

GitHub Actions

concurrency:
  # A new push to the same branch cancels the run that is
  # already in flight. Nobody needs results for a commit
  # that has been superseded.
  group: ci-${{ github.ref }}
  cancel-in-progress: true

jobs:
  static:
    runs-on: ubuntu-latest
    timeout-minutes: 5          # a hung job is worse than a failed one
    steps:
      - uses: actions/checkout@v4   # pin to a full commit SHA in production
      - run: shellcheck scripts/*.sh
      - run: hadolint Dockerfile

  test:
    needs: static               # will not start until static passes
    runs-on: ubuntu-latest
    timeout-minutes: 15
    strategy:
      fail-fast: false          # see note below
      matrix:
        suite: [unit, contract]
    steps:
      - uses: actions/checkout@v4
      - run: make test-${{ matrix.suite }}

That fail-fast: false looks like it contradicts the whole post, so it is worth explaining. Matrix fail-fast defaults to true: one leg fails, the rest get cancelled. That is right when the legs are redundant and you only need to know that something broke. It is wrong when the legs tell you different things. If your matrix is three runtime versions and only one fails, cancelling the others hides whether this is a version-specific problem or a general one, and you will burn a rerun finding out. Fail fast on the pipeline, not on the diagnosis.

timeout-minutes is the setting people skip and regret. A job waiting on an interactive prompt, a dead package mirror, or a container that never becomes healthy will sit there consuming a runner until someone notices. Set a timeout on every job, generous enough that a slow day does not trip it and tight enough that a hang gets caught inside one coffee break.

GitLab CI

stages: [static, test, build, deploy]

default:
  interruptible: true     # allows auto-cancel when a newer pipeline starts
  retry:
    max: 2                # capped at 2 by GitLab, so 3 attempts total
    when:
      # Retry only on infrastructure problems.
      # Never retry script_failure: that is your code failing,
      # and retrying it just hides a flaky test.
      - runner_system_failure
      - stuck_or_timeout_failure
      - api_failure

lint:
  stage: static
  timeout: 5m
  script:
    - shellcheck scripts/*.sh

integration:
  stage: test
  needs: [lint]           # DAG: starts as soon as lint passes
  script:
    - make integration

The retry: when list is the important bit and it is the one people get lazy about. A bare retry: 2 retries on everything including a genuine test failure, which converts an intermittent bug into an invisible one. Scope retries to the failure classes that are genuinely the infrastructure’s fault. Jenkins has failFast true for parallel stages, Buildkite and CircleCI have their own equivalents, and the reasoning transfers cleanly.

A failure should hand you the answer, not the search

Here is the test I apply to a pipeline failure. Can someone who did not write the change, at an inconvenient hour, fix it from the failure output alone without opening the source? If not, the failure message is incomplete.

Four things need to be in there:

• What failed, named precisely. Not “build failed” but the command and its exit code.
• Why, in terms of the input rather than the internals. “lockfile is out of date with package.json”, not “npm ci exited 1”.
• Where, meaning the file, the line, or the config key that caused it.
• What to do next. One concrete action, even if it is just a runbook link.

In shell, a small helper plus an ERR trap gets you most of the way there.

#!/usr/bin/env bash
set -euo pipefail

fail() {
  echo "ERROR: $1" >&2
  echo "  fix: $2" >&2
  exit 1
}

# Catches anything you did not write an explicit check for,
# and tells you the line and the command that blew up.
trap 'echo "FAILED at line $LINENO: $BASH_COMMAND" >&2' ERR

[[ -f config/app.yaml ]] || fail 
  "config/app.yaml is missing" 
  "copy config/app.yaml.example and set DATABASE_URL"

[[ -n "${IMAGE_TAG:-}" ]] || fail 
  "IMAGE_TAG is empty" 
  "check the tag job output; it sets IMAGE_TAG from the git SHA"

Note ${IMAGE_TAG:-} rather than $IMAGE_TAG. Under set -u the bare form aborts with a shell error before your nice message ever prints, which is technically correct and practically useless.

On GitHub Actions you can go one step further and attach the error to the file and line, so it shows up as an annotation on the pull request instead of something to be dug out of a log.

# Renders as a red annotation on that exact line in the diff view.
echo "::error file=deploy/values.yaml,line=42::image tag is empty"

# Visible, but does not fail the job.
echo "::warning file=Dockerfile,line=7::apt cache not cleaned"

Two more habits worth building. Upload logs, test reports, and screenshots as artifacts on failure, conditionally, so the evidence survives the job that produced it. And emit test results as JUnit XML so the platform renders a list of failed test names instead of asking people to scroll. Both are cheap and both cut the time between “it broke” and “I know why” more than any clever retry logic will.

Exit codes are a contract, so use more than one

Most scripts have two states: 0 and “something went wrong”. That collapses distinctions the pipeline needs. Terraform gets this right and it is a good model to copy.

# -detailed-exitcode changes the meaning of the exit code:
#   0 = succeeded, no changes
#   1 = error
#   2 = succeeded, changes present
set +e
terraform plan -detailed-exitcode -out=tfplan
code=$?
set -e

case "$code" in
  0) echo "No drift. Skipping apply." ;;
  2) echo "Changes detected. Holding for approval." ;;
  *) echo "Plan failed. See log above." >&2; exit "$code" ;;
esac

The set +e around it is deliberate: with -e active, exit code 2 would kill the script before you got to interpret it. This is one of the few places where turning the flag off is the correct move, and it is worth a comment in the file explaining why so nobody “fixes” it later.

Do the same in your own scripts. Reserve a code for “the thing you asked for is not possible” versus “I could not reach the API” versus “your input is wrong”. The pipeline can then retry one and refuse to retry the others, which is the difference between a retry that absorbs a runner hiccup and a retry that hides a race condition for six months.

Gate on what you care about, not what is easy to check

This is the deploy-stage version of the same lie. Plenty of commands confirm that a request was accepted, not that the thing you wanted happened.

# Returns 0 the moment the API server accepts the manifest,
# even if every pod goes into CrashLoopBackOff a second later.
kubectl apply -f k8s/

# This is the step that actually waits for the rollout,
# and fails the job if it does not converge.
kubectl rollout status deployment/api --timeout=180s

# And this proves the application answers a request,
# not just that a pod reports Ready.
curl --fail --silent --show-error https://api.example.com/healthz

Run that last check against the public hostname, through your CDN, rather than against the pod directly. If you front the app with Cloudflare or similar, a smoke test that skips the edge will happily pass while real users hit a cached error page or a misrouted origin. Same reasoning applies to the origin host itself: if the app runs on a VPS from a provider like InterServer behind a reverse proxy, verify through the proxy, because that is the path traffic takes.

Then track how long each stage takes over time. Pipeline duration and failure rate per stage are perfectly ordinary metrics to ship into Prometheus and graph in Grafana, or into whatever platform you already use for application monitoring. When the fast lane creeps from forty seconds to four minutes, that is a slow regression nobody will notice from a single run.

Troubleshooting

The job passed but the artifact is wrong

Look for pipes first. Grep the scripts for | and check whether pipefail is set in that shell. Then check for tools that return 0 on failure: curl without --fail, docker run where the entrypoint swallows errors, anything wrapped in || true that somebody added during an incident and never removed.

set -e is set but the script keeps going

The failing command is almost certainly inside a condition, on the left of || or &&, or in a function that is itself being called from an if. Add set -x temporarily and read the trace to see exactly where control flow continued. Then use an explicit check with your fail helper rather than relying on -e at that point.

Works locally, fails in CI (or the reverse)

Usually a shell difference. Your terminal runs bash or zsh, the runner may execute the script with sh. Add an explicit shebang, and on GitHub Actions set the shell on the step or at the job level rather than assuming. Inside containers, remember dash has no pipefail and no [[ ]].

The job hangs until the platform kills it

Something is waiting on stdin. Package managers prompting for confirmation, ssh asking about a host key, a database client waiting for a password. Add the non-interactive flags, set CI=true where tools respect it, and put a timeout on the job so the next occurrence fails in ten minutes instead of sixty.

Fails on the first run, passes on rerun

Resist the retry. Log the failure with enough context to identify it later, quarantine the test into a non-blocking job if it is genuinely blocking the team, and give it an owner and a date. A blanket retry turns a flaky test into a permanent tax on everyone’s confidence in the pipeline.

The failure output is a wall of noise

Suppress progress bars and spinners in CI, they render as thousands of unreadable lines. Then print a short summary block at the very end of the job, after the noise, listing what failed and what to do. People read the bottom of a log first.

Common mistakes

• Writing set -e without -o pipefail and assuming the script is covered.
• Using || true to silence a step that fails intermittently, instead of finding out why.
• Running the slowest, flakiest checks before the ones that take fifteen seconds.
• Retrying on every failure class rather than only infrastructure faults.
• No job timeouts, so a hang consumes a runner until someone notices.
• Treating kubectl apply or a deploy API call as proof the deploy worked.
• Failure messages that name the internal function that threw, and nothing the reader can act on.
• Caching so aggressively that a stale dependency tree hides a real break.
• Discarding logs and test reports when the job fails, which is exactly when you need them.
• Turning off matrix fail-fast everywhere by reflex, so every broken commit costs the full matrix.

Best practices

• Put set -euo pipefail at the top of every shell script the pipeline runs, and set the Dockerfile SHELL before any piped RUN.
• Order stages so each one is cheaper and more likely to fail than the next.
• Give every job a timeout, and cancel superseded runs on the same branch.
• Use explicit checks with actionable messages instead of relying on -e to catch everything.
• Distinguish exit codes so the pipeline can react differently to different failures.
• Verify deploys with a rollout status check plus a real HTTP request through the public path.
• Upload artifacts and structured test reports on failure.
• Pin action and image references to immutable digests so a green pipeline stays reproducible.
• Track stage duration and failure rate over time, the same way you track application metrics.
• Fix the failure message the first time somebody has to ask what it means.

FAQ

What does “fail fast” actually mean in a CI/CD pipeline?

Two things. First, the pipeline stops as soon as it knows the result will be a failure, rather than finishing work whose outcome no longer matters. Second, the cheapest checks that are most likely to catch a problem run first, so the average failure surfaces in seconds. The second half is a design choice about ordering, not a platform setting.

Is set -e enough on its own?

No. It is suspended inside conditions, on the left side of && and ||, and after !. It also does nothing about pipes, which is why -o pipefail matters, or about commands that return 0 while failing, which is why explicit checks matter. Use it as a baseline, not as your only line of defence.

Should I disable matrix fail-fast?

Only when the legs give you different diagnostic information. If the matrix is runtime versions or operating systems and you need to know whether the break is universal or specific, let all legs run. If the legs are interchangeable shards of the same test suite, leave fail-fast on and save the compute.

How do I stop a pipeline from passing when a step really failed?

Work through three things in order. Set pipefail so pipes propagate failure. Add the flags that make tools return non-zero on failure, such as curl --fail. Then replace implicit trust with explicit assertions: after a deploy, check that the rollout converged and that the service answers a request. Most silent passes come from one of those three gaps.

Are automatic retries a good idea?

Only for failures that are the infrastructure’s fault: runner crashes, image pull errors, API timeouts from the CI platform itself. Never retry a script failure by default. And make retries visible, because a job that quietly succeeds on the third attempt is telling you something important that nobody sees.

How long should a fast feedback stage take?

Short enough that an engineer waits for it rather than switching context. Somewhere under a couple of minutes for the combined lint, format, and unit stage is a reasonable target for most codebases. Past that, people push and walk away, and you lose the benefit of fast feedback even though the pipeline is technically quick.

Does this apply to self-hosted runners too?

All of it, and timeouts matter more. On hosted runners a stuck job eventually hits the platform’s own ceiling. On your own machine it holds a slot until you intervene, so a hung job blocks the queue for everyone. Set explicit timeouts and make sure the runner cleans up its workspace between jobs.

The one thing to take away

Speed is the part everyone optimises and honesty is the part that actually costs money. A fail fast pipeline that finishes in ninety seconds is worth very little if one of those seconds contains a swallowed exit code. Fix the lying first: pipefail, tools that return non-zero when they fail, and assertions that check the outcome rather than the request. Then reorder for speed.

And judge every failure by one question. Could someone else fix this from the output alone, without reading the source? If the answer is no, the pipeline has not finished its job, no matter how quickly it went red.

Need help fixing a pipeline that lies to you?

If any of this sounds like your CI, this is the kind of work I take on. Typical engagements look like:

• Auditing an existing GitHub Actions, GitLab CI, or Jenkins pipeline for silently swallowed failures and false-green stages.
• Restructuring stage order and caching so the common failure surfaces in seconds instead of after the build.
• Hardening deploy verification: rollout gates, smoke tests through the real public path, and automatic rollback on a failed health check.
• Rewriting pipeline shell scripts with proper error handling, exit code contracts, and failure messages an on-call engineer can act on.
• Wiring pipeline duration and failure metrics into Prometheus and Grafana so regressions get caught early.

Send me the workflow file and a link to a failing run, and I will tell you what I would change first.

The post Green Check, Broken Build: How to Build a Fail Fast Pipeline That Explains Why appeared first on John Nessime.

None of that is a bug. AWS optimises the empty account for getting something running in ten minutes. The problem is that most accounts never get corrected afterwards, and eighteen months later there is production traffic sitting on top of decisions nobody made deliberately.

An AWS account baseline is the set of things you lay down before the first workload, so the account starts from a defensible posture instead of a blank console. This walks through the decisions that actually matter, why each one is the way it is, and what it costs. I have put a working reference implementation on GitHub — terraform-aws-baseline — so you can read the real Terraform rather than take my word for any of it.

What a baseline is, and what it is not

A baseline is not a landing zone in the Control Tower sense, and it is not a compliance framework. It does not make you SOC 2 compliant and nothing that fits in one repository will.

What it is: the boring foundational layer that everything else sits on. State storage, network segmentation, identity, and the guardrails that stop a careless afternoon becoming an incident. If you get these right once, every workload that lands afterwards inherits them for free.

The structure that works is three layers. A bootstrap stack that creates the state backend and runs exactly once. Reusable modules for the things every environment needs. Thin per-environment roots that compose those modules with different parameters — dev gets one NAT gateway and seven-day log retention, prod gets one per availability zone and a year.

Decision one: no long-lived AWS credentials, anywhere

Static access keys in CI are the single most common way AWS accounts get compromised. They live in a secrets store, they get copied into a second one, somebody echoes one into a build log, and nobody rotates them because rotation breaks things.

GitHub Actions can assume an AWS role using a short-lived OIDC token that GitHub signs and AWS verifies. There is no key to store, nothing to leak, and nothing to rotate.

The important part is the trust policy, and one word in it decides whether this is a security improvement or a vulnerability:

data "aws_iam_policy_document" "ci_assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    # StringEquals, not StringLike. This is the whole ballgame.
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values = [
        "repo:${var.github_org}/${var.github_repo}:ref:refs/heads/${var.github_branch}"
      ]
    }
  }
}

Swap StringEquals for StringLike and add an asterisk, and you have widened that trust from one branch of one repository to whatever the wildcard covers. Get sloppy enough — a bare repo:* — and any repository on GitHub can assume your role. This is not theoretical; it is a recurring finding in real audits.

Scope it to one repository and one branch. If you need more, add more statements, explicitly.

Decision two: treat Terraform state as a secret, because it is

Terraform state contains every attribute of every resource it manages. Database passwords, generated keys, connection strings. Marking a variable sensitive redacts it from console output and changes nothing about what is written to the state file.

So the state bucket gets four things: server-side encryption with a customer-managed KMS key that rotates, versioning so a corrupted state can be recovered, all four public access block flags, and a bucket policy that refuses plaintext connections outright.

resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

data "aws_iam_policy_document" "deny_insecure_transport" {
  statement {
    sid       = "DenyInsecureTransport"
    effect    = "Deny"
    actions   = ["s3:*"]
    resources = [
      aws_s3_bucket.state.arn,
      "${aws_s3_bucket.state.arn}/*",
    ]

    principals {
      type        = "*"
      identifiers = ["*"]
    }

    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }
}

All four public access block flags, not two. They cover different paths — ACLs and bucket policies, existing objects and future ones — and setting a subset leaves a gap that reads as protected on a dashboard.

There is a chicken-and-egg problem here that catches everyone: the stack that creates the state bucket cannot store its own state in a bucket that does not exist yet. You run it once with local state, then migrate:

cd bootstrap
terraform init
terraform apply
terraform output          # bucket, lock table, KMS key, region

# Now move bootstrap's own state into the bucket it just created
terraform init -migrate-state

Decision three: shut the default security group

Every VPC gets a default security group that permits all traffic between members and all outbound traffic. Anything launched without an explicit security group lands on it. That is a quiet failure mode — nothing errors, the instance works, and it has more network reach than anyone intended.

Terraform can adopt and empty it:

resource "aws_default_security_group" "default" {
  vpc_id = aws_vpc.this.id

  # No ingress and no egress blocks. Terraform strips AWS's rules,
  # so anything that accidentally lands here can reach nothing.
}

Four lines, and misconfiguration becomes loud instead of silent.

Decision four: three subnet tiers, and the bottom one has no way out

The standard two-tier split — public and private — leaves databases in the same tier as application servers. Both can reach the internet through NAT. That is fine right up until something in the data tier is compromised and quietly exfiltrates over 443, which looks like ordinary egress traffic.

• Public — NAT gateways and load balancers. Route to the internet gateway.
• Private — workloads. Outbound through NAT, no inbound from the internet.
• Database — data stores. No default route at all.

The third tier is enforced by absence rather than by a rule:

resource "aws_route_table" "database" {
  vpc_id = aws_vpc.this.id

  # Deliberately no 0.0.0.0/0 route. Not to the IGW, not to NAT.
  # Nothing here can reach the internet, and nothing can reach it.
}

No security group to misconfigure, no NACL to get wrong. The route simply does not exist. That is a much stronger guarantee than a rule someone can widen during an incident and forget to narrow afterwards.

Decision five: gateway endpoints, the one genuinely free win

S3 and DynamoDB gateway endpoints cost nothing. Not “cheap” — zero. They add a route so traffic to those services stays on the AWS backbone rather than going out through NAT.

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.this.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"

  route_table_ids = concat(
    aws_route_table.private[*].id,
    aws_route_table.database[*].id,
  )
}

Two benefits, and the second is the one people miss. Traffic to S3 never traverses NAT, so you stop paying per-gigabyte processing on it. And the isolated database tier can still reach S3 and DynamoDB despite having no internet route, because the endpoint is a route within the VPC rather than a path out of it.

Your Terraform state lives in S3. Without this endpoint, every plan and apply pushes state through the NAT gateway and you are billed for the privilege.

Decision six: SSM Session Manager, not SSH

Port 22 open anywhere means key pairs to distribute, a bastion host to patch, and an audit trail that is whatever auth.log happens to contain.

resource "aws_iam_role_policy_attachment" "ssm_core" {
  role       = aws_iam_role.instance.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

That one attachment gets you shell access with no inbound ports, no key pairs, no bastion. Access is IAM-authorised, so revoking someone is a permissions change rather than a key rotation, and every session can be logged centrally.

Worth knowing the caveat: SSM agents reach the SSM endpoints over the internet by default, so instances in the private tier still need NAT for this. Removing NAT entirely means adding interface endpoints for ssm, ssmmessages and ec2messages, and those cost roughly seven dollars a month per availability zone each.

Decision seven: pin actions to commit SHAs

A Git tag is a movable pointer. If an action’s repository is compromised, an attacker can repoint v4 at whatever they like, and every workflow referencing that tag runs it on the next build — with your OIDC role attached.

steps:
  # A tag can be moved. A commit SHA cannot.
  # Dependabot updates these pins and shows you the diff.
  - uses: actions/checkout@<full-40-character-sha>   # v4.2.2

The comment matters as much as the pin. Without it nobody can tell at a glance which version they are looking at, and the pin becomes something people update blindly.

What it actually costs

Almost all of it is NAT gateways. Everything else is rounding.

• NAT gateways — roughly $32 a month each, plus about $0.045 per gigabyte processed. One in dev, one per availability zone in prod.
• VPC, subnets, route tables, internet gateway — free.
• S3 and DynamoDB gateway endpoints — free.
• KMS keys — about a dollar a month each.
• Flow logs — around $0.50 per gigabyte ingested. Retention is the lever: seven days in dev, a year in prod.
• State bucket and lock table — cents.

Ballpark, that lands around $35 a month for dev and $105 for prod, dominated entirely by NAT. If you only need the isolated tiers and nothing requires outbound internet, dropping NAT takes dev close to free.

Per-AZ NAT in production is an availability decision, not a security one. A single NAT gateway is a single point of failure for outbound traffic across every AZ that routes through it.

Gates that run before any credentials exist

Most of what goes wrong in Terraform is catchable statically. Formatting, syntax, provider misuse, and known-insecure patterns can all be found without touching an AWS account, which means they can run on every commit and in CI without the pipeline holding any cloud credentials at all.

# On every commit, locally
pre-commit install        # gitleaks, fmt, validate, tflint, checkov, terraform-docs

# The same gates CI runs
make all                  # fmt + validate + tflint + checkov

Running the identical command locally and in CI is the part that matters. When those two drift, people stop trusting the local check and start pushing to see what the pipeline says, and the fast feedback loop you built quietly stops existing.

Note also what is missing: there is no terraform plan in that pipeline. Plan needs real credentials, and a static-only pipeline is one that cannot leak anything. A production setup would add a plan job running under the OIDC role and posting to the pull request — but that is a deliberate step up in blast radius, not a default.

What a baseline like this deliberately leaves out

Being clear about scope is more useful than pretending completeness. The reference implementation covers preventative controls and stops there. Missing, on purpose:

• Detective controls. No CloudTrail, Config, GuardDuty or Security Hub. A complete account baseline includes them; this is the preventative half.
• Compute. There is an instance role and profile, but nothing uses them. No ASG, no ECS or EKS, no load balancer.
• Interface endpoints. Only the free gateway ones. Fully private SSM needs paid interface endpoints.
• Multi-region and DR. Versioning covers state recovery. There is no cross-region replication and no failover story.
• Policy-as-code tests. Checkov catches known misconfigurations; there are no Terratest or Conftest assertions on module contracts.

A baseline you can describe the edges of is more useful than one advertised as complete. You know exactly what you still have to build.

Common mistakes

• Using StringLike with a wildcard in an OIDC trust policy. Scope to one repository and one branch.
• Assuming sensitive = true keeps values out of state. It does not.
• Setting two of the four S3 public access block flags and considering the bucket locked down.
• Leaving the default security group as AWS shipped it.
• Putting databases in the same tier as application servers because two tiers felt like enough.
• Skipping gateway endpoints and paying NAT processing charges on your own state file.
• Referencing actions by tag and assuming a tag is immutable.
• Committing backend.hcl or terraform.tfvars. State bucket names embed the account ID.

Frequently asked questions

Is an AWS account baseline the same as a landing zone?

No. A landing zone in the Control Tower sense governs many accounts — organisational units, service control policies, account vending. A baseline is what you lay down inside a single account. Baselines are what you need when you have one or two accounts and Control Tower would be enormous overkill.

Why not just use Control Tower?

Control Tower is a good answer for a multi-account organisation with governance requirements. It is a lot of machinery for a team running a handful of accounts, and it hides the decisions rather than making them legible. Writing the baseline yourself means you can read every choice and change any of them.

Do I really need a separate database subnet tier?

It costs nothing and closes an entire exfiltration path. If your data tier has no route to the internet, a compromised database cannot phone home regardless of what the attacker does with it. Two tiers is not wrong, but the third is close to free.

How do I cut the NAT gateway cost?

Use one NAT for non-production instead of one per AZ. Add gateway endpoints so S3 and DynamoDB traffic bypasses NAT entirely. Beyond that, work out what genuinely needs outbound internet — often less than you assume — and consider dropping NAT for environments that do not.

Can I use this with OpenTofu?

Yes. It is standard HCL against the AWS provider, so it runs on either. Swap terraform for tofu in the commands and everything else is the same.

Where does the OIDC provider itself live?

It is account-global, so only one environment can create it and the others reference it. That is workable but awkward. In a real organisation it belongs in a separate account-level stack that runs before any environment.

Wrapping up

None of the decisions in an AWS account baseline are individually clever. Scope the trust policy. Encrypt the state. Empty the default security group. Give the data tier no way out. Take the free endpoints. Use SSM instead of SSH. Pin your actions. Each one is a few lines of Terraform.

What makes them valuable is doing them before the first workload arrives, because retrofitting network segmentation into a running account is a project, and doing it at the start is an afternoon.

The full implementation is on GitHub at JohnNessime/terraform-aws-baseline, MIT licensed. It is a reference implementation rather than something that has served production traffic — a worked example of how I would lay an account down, with the reasoning written out in the docs. Read it, disagree with parts of it, take what is useful.

Need a baseline laid down properly?

I work with teams on AWS foundations — account structure, network segmentation, IAM that does not rely on long-lived keys, Terraform layouts that survive more than one engineer, and CI pipelines that catch problems before they reach an account.

If you would rather have this done once and done right, you can hire me on Upwork.

The post A Secure-by-Default AWS Account Baseline in Terraform appeared first on John Nessime.

The Pulumi vs Terraform question got harder in the last two years, not easier. The licence changed, the community forked, IBM bought HashiCorp, HashiCorp killed its own general-purpose-language option, and Pulumi started speaking HCL. A lot of the comparison articles you will find were written before any of that happened.

This is an attempt at an honest read of where things actually stand, what the real trade-off is once you strip away the marketing, and how to pick a lane you will not regret in eighteen months.

What actually changed

A short timeline, because the current landscape only makes sense against it.

• August 2023 — HashiCorp relicensed Terraform from MPL 2.0 to the Business Source License. Source-available, free for provisioning your own infrastructure, but you cannot build a competing commercial product on it.
• January 2024 — OpenTofu shipped 1.6.0, a fork of the last MPL-licensed code, governed by the Linux Foundation.
• February 2025 — IBM completed its acquisition of HashiCorp, valued at around $6.4 billion.
• December 2025 — HashiCorp deprecated CDK for Terraform and archived the repository. The official way to write Terraform in a general-purpose language no longer exists.
• Also December 2025 — Pulumi announced native HCL support and the ability to work with Terraform and OpenTofu state.

Those last two matter more than the licence drama, and they point in opposite directions. HashiCorp closed the door on writing Terraform in a real language. Pulumi opened the door to writing Pulumi in HCL. Whatever else you conclude, “HCL versus real languages” is no longer a clean line between two products.

The real axis: a configuration language or a programming language

Strip out the licensing and the corporate ownership and one genuine difference remains. HCL is a purpose-built configuration language that describes desired state. Pulumi runs an actual program that constructs a resource graph, and then applies it.

Here is the same trivial thing in both. Terraform:

resource "aws_s3_bucket" "logs" {
  for_each = toset(var.environments)
  bucket   = "acme-logs-${each.key}"

  tags = {
    Environment = each.key
    ManagedBy   = "terraform"
  }
}

And Pulumi, in TypeScript:

import * as aws from "@pulumi/aws";

const environments = ["dev", "staging", "prod"];

const buckets = environments.map(env =>
  new aws.s3.Bucket(`logs-${env}`, {
    bucket: `acme-logs-${env}`,
    tags: {
      Environment: env,
      ManagedBy: "pulumi",
    },
  })
);

For something this small the difference is aesthetic. It stops being aesthetic around the third nested conditional.

What you actually gain with a general-purpose language

• Real abstractions. Classes, functions, inheritance. A Pulumi component that encapsulates “our standard service” is just a class, and it composes the way code composes.
• Real testing. Jest, pytest, Go’s testing package. You can unit test infrastructure logic with the same tools and the same CI as the rest of your codebase.
• The package ecosystem. Need to parse a CSV of subnets, call an internal API, or do something genuinely awkward? It is an npm install rather than an external data source and a shell script.
• IDE support that works. Type checking, refactoring, jump-to-definition. HCL tooling has improved a lot but it is not the same thing.

What you lose, which comparison posts skip

This is the part worth reading twice, because it is where the regret comes from.

• Constraint is a feature. HCL is limited on purpose. When a language cannot express clever things, nobody writes clever things. Give a team a real language and someone will eventually build a metaprogramming layer that generates resource names from a database lookup, and reviewing that pull request will be nobody’s idea of a good afternoon.
• Readability across skill levels. A network engineer who has never written TypeScript can usually read HCL and tell you what it provisions. The reverse is not true. If your infrastructure is touched by people who are not primarily programmers, that matters enormously.
• The hiring pool. There are far more engineers with Terraform on their CV than Pulumi. That is not an argument about quality, it is an argument about how long it takes to backfill a role.
• A dependency on a runtime. Pulumi programs need Node, Python, or Go present and correct in every environment that runs them, including CI. It is one more thing that can drift.
• Asynchrony leaks into the model. Resource outputs are not plain values. They are futures, and you have to handle them as such. Every Pulumi team hits this on day two.

# This does not do what a newcomer expects. bucket.id is an Output,
# not a string, so you get an object reference in the name.
# bucket_name = f"logs-for-{bucket.id}"

# Correct: work inside apply, where the value is resolved.
bucket_name = bucket.id.apply(lambda bid: f"logs-for-{bid}")

None of these are dealbreakers. They are just the bill that arrives later, and it is worth reading before you sign.

State, providers, and the things that actually break

State management

Both tools keep a state file mapping your code to real resources, and both will ruin your afternoon if that file gets corrupted or out of sync.

Terraform and OpenTofu use pluggable backends: S3 with DynamoDB locking, Azure Blob, GCS, Consul, or HCP Terraform. OpenTofu added client-side state encryption, which Terraform does not have natively and which is a genuine advantage if your compliance people care about what is sitting in that bucket.

Pulumi defaults to Pulumi Cloud, which is free for individuals and paid for teams. It also supports self-managed backends, and you should know this because the default sometimes gets mistaken for a requirement:

# Self-managed state in S3, no Pulumi Cloud account involved
pulumi login s3://my-pulumi-state-bucket

# Or entirely local, useful for experiments
pulumi login --local

Pulumi encrypts secrets in state by default. With Terraform, values marked sensitive are redacted in output but still sit in plaintext in the state file, which surprises people who assumed otherwise.

The provider ecosystem, and a myth worth killing

You will read that Pulumi has a much smaller provider ecosystem. That was true once and is mostly no longer the point, because Pulumi bridges Terraform providers. If a Terraform provider exists, you can almost certainly use it from Pulumi, either pre-built in the Pulumi Registry or generated on demand.

Two caveats that are real:

• Bridged providers can lag upstream releases. If you need a resource type that shipped last week, check before committing.
• When something breaks deep in a bridged provider, you are debugging across a translation layer. That is a worse afternoon than debugging the provider directly.

Pulumi also ships native providers for Kubernetes and Azure, generated from the platform APIs, which give same-day coverage of new resources rather than waiting on a bridge.

Policy and governance

Terraform’s first-class answer is Sentinel, which is tied to HCP Terraform. OpenTofu users generally reach for OPA or Conftest. Pulumi has CrossGuard, where policies are written in the same languages as everything else.

If you are in a regulated environment and your auditors already know what Sentinel is, that is a heavier thumb on the scale than any language preference.

Licensing, since it is what forced the question

• Terraform — BSL 1.1. Source-available. Fine for provisioning your own infrastructure. Not fine for building a product that competes with HashiCorp. Each release converts to an open licence four years after publication, but only that release.
• OpenTofu — MPL 2.0, OSI-approved, Linux Foundation governance. No single vendor can relicense it.
• Pulumi — Apache 2.0 for the engine and CLI. Permissive, with a patent grant. Pulumi Corp monetises through Pulumi Cloud rather than through the licence.

For the overwhelming majority of teams, the BSL changes nothing about daily work. You are provisioning your own infrastructure and that is explicitly permitted. The concern is not today’s terms, it is that a single vendor demonstrated it can change them, and that vendor is now part of IBM.

If your objection to Terraform is licensing rather than language, the answer is OpenTofu, not Pulumi. Switching paradigms to solve a legal problem is an expensive way to do it.

Choosing a lane

Three coherent positions. Pick the one that matches your constraints rather than the one that sounds most modern.

Lane one: OpenTofu

The default for most teams already running Terraform who are uneasy about the licence. Migration is genuinely close to free: same HCL, same providers, same state format, same commands.

# Back up state first. Always.
terraform state pull > backup-$(date +%F).tfstate

# Then initialise with tofu against the same backend
tofu init
tofu plan

A clean plan with no proposed changes means you are done. That is the whole migration for most codebases.

Choose this if: you have existing Terraform, HCL is working for you, and you want open governance without retraining anyone.

Lane two: Terraform with HCP

Still the biggest ecosystem, the most documentation, the most Stack Overflow answers, the most people who already know it. Stacks and ephemeral resources are Terraform-only. Sentinel is Terraform-only. If you are deep in HCP Terraform and it works, the licence is an abstract concern rather than a practical one.

Choose this if: you are in a regulated industry, you need vendor support with a contract behind it, or your organisation is standardising on IBM and Red Hat tooling anyway.

Lane three: Pulumi

The strongest case for Pulumi is not “real languages are nicer”. It is when your infrastructure genuinely needs logic that HCL fights you on: dynamic topologies, infrastructure derived from application code, platform teams building abstractions for other teams to consume. If you are writing a self-service platform rather than a set of environments, a real language stops being a preference and starts being the right tool.

Choose this if: your team is primarily software engineers, your infrastructure has real branching logic, or you are building a platform product rather than provisioning a fixed estate.

Do not choose it because writing loops in HCL annoyed you once.

What migrating actually looks like

Terraform to OpenTofu is a swap. Terraform to Pulumi is a refactor, and anyone telling you otherwise is selling something.

Pulumi does provide real tooling for it:

# Convert HCL into a Pulumi program in your chosen language
pulumi convert --from terraform --language typescript

# Adopt the resources described by an existing state file
pulumi import --from terraform ./terraform.tfstate

Pulumi can also consume Terraform modules directly, so existing module investment is not automatically wasted. That is the single most useful fact for anyone considering a gradual move.

A sane sequence:

1. Pick one small, low-risk stack. Not the production VPC.
2. Convert it, import the state, and run a preview. The only acceptable result is no changes.
3. Refactor the generated code. Converted output is mechanically correct and stylistically awful, which is fine, but do not leave it that way.
4. Run both tools side by side for a while. Coexistence is supported; a big-bang cutover is not necessary.
5. Only then decide whether to move anything else.

If step two produces a diff you cannot explain, stop. An unexplained diff during an IaC migration is how resources get destroyed and recreated.

Common mistakes

• Switching tools to solve a licensing concern when OpenTofu solves it for free.
• Picking the tool the loudest engineer prefers rather than the one the whole team can maintain.
• Adopting CDKTF in 2026. It was archived in December 2025 and will not receive fixes.
• Treating a converted program as finished code rather than a starting point.
• Assuming Terraform state is safe because values are marked sensitive. Encrypt the backend.
• Migrating production first because it is the stack that matters most. It is the stack that matters most.
• Running two tools against the same resources without a clear ownership boundary.

Best practices whichever lane you pick

• Remote state with locking and encryption, from the first commit. Never local state for anything shared.
• Small stacks with clear boundaries. One enormous state file is the most common self-inflicted wound in this field.
• Plan or preview output in every pull request, and require a human to read it.
• Pin provider versions. Floating versions turn an unrelated deploy into an incident.
• Policy as code before you need it, not after the audit.
• Document the decision. Six months from now nobody will remember why, and the reasoning is worth more than the choice.

Frequently asked questions

Is Pulumi better than Terraform?

Neither is better in the abstract. Pulumi suits teams of software engineers building infrastructure with real logic in it. Terraform and OpenTofu suit teams provisioning a defined estate where constraint and readability matter more than expressiveness.

Can I use Terraform providers with Pulumi?

Yes. Pulumi bridges Terraform providers, either pre-built in its registry or generated on demand, and can consume Terraform modules directly. Bridged providers can lag upstream releases, so verify coverage for anything recent.

Should I move from Terraform to OpenTofu?

If licensing is your concern, it is close to free: same HCL, same providers, same state. Back up state, run tofu init and tofu plan, and confirm no changes. If you rely on Stacks, Sentinel, or HCP Terraform, staying put is reasonable.

What happened to CDKTF?

HashiCorp deprecated it on 10 December 2025 and archived the repository. It gets no further fixes or compatibility updates. If you want general-purpose languages for infrastructure now, Pulumi is the maintained option.

Does the BSL licence stop me using Terraform commercially?

Not for provisioning your own infrastructure, which is what almost everyone does. It restricts building a competing commercial IaC product on top of it. Check with your own legal team rather than a blog post if you are anywhere near that line.

Can Pulumi and Terraform coexist?

Yes, and for any migration of size that is the sensible approach. Pulumi can reference Terraform state and consume Terraform modules. The rule is that each resource has exactly one owner.

Wrapping up

The Pulumi vs Terraform decision is less about the tools than about who maintains your infrastructure and what it has to do. If your estate is a known set of environments maintained by people with mixed backgrounds, a constrained configuration language is a feature and OpenTofu gives you that without licensing anxiety. If you are building a platform, with branching logic and abstractions other teams consume, a real language earns its complexity and Pulumi is the maintained way to get one.

What has genuinely changed is that the lanes are no longer walled off. Pulumi speaks HCL and reads Terraform state. OpenTofu is a drop-in for Terraform. Converting between them is tooling rather than a rewrite. That makes the decision less permanent than it feels, which is a good reason to pick deliberately and a bad reason to agonise.

Pick the lane your team can still maintain when the person who chose it has moved on. Then write down why.

Need help choosing or migrating?

I work with teams on infrastructure as code decisions and the migrations that follow — Terraform to OpenTofu, Terraform to Pulumi, splitting up state files that grew too large, and wiring the whole thing into CI with policy checks that actually run.

If you would rather talk it through with someone who has untangled a few of these, you can hire me on Upwork.

The post Pulumi vs Terraform: Choosing a Lane appeared first on John Nessime.