That failure should shape how you think about EKS Pod Identity vs IRSA. Both deliver the same end result to your application: short-lived, rotated credentials with nothing hardcoded. The differences are in how trust is expressed, who has to touch IAM to change it, and what happens when the wiring is subtly wrong.

This post compares the two on what actually decides the choice: how each scales across clusters, where each refuses to work at all, the failures each hides, and a decision procedure you can apply to a real cluster.

Same outcome, two very different trust shapes

Most of the trade-offs fall out of the mechanism, so it is worth being precise about both.

IRSA: federation through an OIDC provider

The cluster has an OIDC issuer, which you register as an identity provider in IAM. A projected service account token is mounted into the pod, and the SDK exchanges it for credentials through the web identity flow against STS. The role’s trust policy is what makes that work, and it names one specific cluster:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub":
          "system:serviceaccount:production:my-app",
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud":
          "sts.amazonaws.com"
      }
    }
  }]
}

The important detail is that the issuer URL is unique per cluster. The relationship between a workload and a role lives inside IAM, and it is cluster-specific by construction.

Pod Identity: an EKS API and an on-node agent

Pod Identity moves that mapping out of IAM. You install the EKS Pod Identity Agent add-on, which runs as a DaemonSet, then create an association through the EKS API binding a namespace plus service account name to a role. The agent calls the EKS Auth API for temporary credentials, and the pod picks them up through the container credential provider that AWS SDKs already support.

The trust policy contains nothing cluster-specific:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
    "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": [ "sts:AssumeRole", "sts:TagSession" ]
  }]
}

sts:TagSession is not decoration. Pod Identity attaches session tags on every assume, so a policy allowing only sts:AssumeRole fails. It is the most common first-attempt error.

Creating the association is a single call:

aws eks create-pod-identity-association 
  --cluster-name prod-eu 
  --namespace production 
  --service-account my-app 
  --role-arn arn:aws:iam::111122223333:role/my-app

Where IRSA still wins

Treating IRSA as legacy is the most common mistake here. For some workloads it is not the older option, it is the only option.

• Fargate pods. The Pod Identity Agent is a DaemonSet on EC2 nodes. Fargate has no node for it to land on, and AWS documents Fargate pods as unsupported. If part of your workload runs on Fargate, IRSA is the mechanism there, permanently, not as a stopgap.
• Windows nodes. Also documented as unsupported for Pod Identity.
• Older SDKs you cannot rebuild. The web identity flow has been in the SDKs far longer, so a vendor image you cannot patch is more likely to work with IRSA.
• Direct federation into workload accounts. If your model wants the pod’s identity federating straight into a role in the resource-owning account, IRSA does that natively. Pod Identity reaches other accounts by role chaining, a different trust shape even when the outcome looks the same.
• You already have it working at a scale you control. A few clusters with a Terraform module that provisions the provider and the roles is not a problem that needs solving.

Where IRSA hurts

The pain is administrative, and it grows with cluster count rather than workload count. Every new cluster brings a new OIDC issuer, which means editing the trust policy of every role those workloads use. Trust policies have a size limit, so a role trusting many issuers eventually runs out of room and you end up duplicating roles to work around it.

The same asymmetry shows up in blue/green cluster upgrades. Standing a new cluster beside the old one means every role used by every migrating workload needs a trust policy edit before the new cluster can do anything, and another edit afterwards to clean up. That is IAM work on the critical path of an upgrade, usually owned by a different team than the one doing it.

Where Pod Identity wins

• Roles become portable. One role, one generic trust policy, any number of clusters. Adding a cluster is an EKS API call per workload, not an IAM edit per role.
• The permission split matches how teams are organised. Creating an association needs EKS permissions, not iam:UpdateAssumeRolePolicy, so cluster admins stop filing IAM tickets for every new service account.
• Session tags arrive for free. Every assume carries eks-cluster-arn, eks-cluster-name, kubernetes-namespace, kubernetes-service-account, kubernetes-pod-name and kubernetes-pod-uid. That is usable attribute-based access control and the CloudTrail audit trail you always wanted.
• Cross-account access without touching application code. An association can carry a target role ARN, and EKS performs the role chaining for you.

The ABAC angle is underrated. One policy can serve dozens of workloads, because the session tag identifies the caller:

"Condition": {
  "StringEquals": {
    "s3:ExistingObjectTag/team": "${aws:PrincipalTag/kubernetes-namespace}"
  }
}

Where Pod Identity hurts

The generic trust policy is a real trade-off. Written exactly as AWS shows it, that role can be assumed on behalf of any service account in any namespace in any cluster in the account, as long as someone with EKS permissions creates an association pointing at it. Blast radius control has moved from IAM into the EKS API. If those permissions are loosely held, you widened it.

The fix is to put the constraint back in the trust policy using the request tags Pod Identity sends:

"Condition": {
  "StringEquals": {
    "aws:RequestTag/kubernetes-namespace": "production",
    "aws:RequestTag/kubernetes-service-account": "my-app"
  }
}

You can also pin to a cluster with aws:SourceArn. That costs some of the portability you switched for, which is the point: decide per role. A role reading one team’s bucket should be pinned. A role used by an identical workload in six clusters should not.

The other cost is a new moving part. The agent is a DaemonSet, so it needs to tolerate your taints, and it must be running before pods on that node can get credentials.

The failures that actually bite

Silent fallback to the node role

This is the one from the opening, and it is invisible by design. Pod Identity works through the container credential provider, which the SDK finds by reading two environment variables EKS injects into the pod:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

An SDK that predates that provider does not error on these. It does not know what they mean, so it keeps walking the credential chain until it reaches instance metadata and picks up the node role. The application works. The permissions are wrong. Nothing complains.

Two defences, and you want both. Check the minimum SDK version list in the EKS documentation and pin base images above it, and block pod access to IMDS so falling back has nowhere to fall. The first prevents the problem; the second turns a silent success into a loud failure.

Verify from inside the pod. The ARN in the response tells you which identity is really in use:

kubectl exec -n production deploy/my-app -- aws sts get-caller-identity

The association that matches nothing

An association is an exact match on cluster, namespace and service account name. A typo in any of the three produces an association that exists, looks fine in the console, and never applies to a pod. List them and read them against your manifests:

aws eks list-pod-identity-associations --cluster-name prod-eu

Associations also only take effect for pods started after they exist. If you create one and nothing changes, restart the workload before you start debugging IAM.

Both mechanisms configured at once

During a migration you will often have a service account carrying an IRSA annotation and a Pod Identity association at the same time. Both paths can work, and which one the SDK uses depends on credential chain ordering. AWS documents that credentials found earlier in the chain keep being used even when an association exists.

Do not reason about it. Check the caller identity and check CloudTrail, then remove the one you are not keeping so nobody has to work it out again.

PackedPolicyTooLarge

Session tags, managed policy ARNs and inline session policies are compressed into a packed format with its own size limit, and six session tags on every assume is not free. The association can disable the automatic tags, at the cost of the ABAC and audit benefits that were half the reason to use Pod Identity.

A decision procedure for EKS Pod Identity vs IRSA

Work through these in order and stop at the first one that answers.

1. Does this workload run on Fargate or Windows nodes? IRSA. There is no decision to make.
2. Can you rebuild the image with a current SDK? If not, IRSA, or fix the image first.
3. Does your model require direct OIDC federation into the resource account? IRSA. Role chaining is not the same shape, even if it reaches the same bucket.
4. Will this role be used from more than one cluster, now or after the next upgrade? Pod Identity. The saving compounds every time you add or replace a cluster.
5. Do you want cluster and namespace in the CloudTrail record without extra work? Pod Identity, because of the session tags.
6. None of the above? Pod Identity for anything new, IRSA left alone where it already works.

AWS’s own guidance lands in the same place: Pod Identity for new applications on supported node types, IRSA where you already have it working or where Pod Identity cannot run.

Arguments that don’t survive contact

• “IRSA is deprecated.” It is not. AWS has said it continues to invest in it, and Fargate support alone guarantees it stays.
• “Pod Identity is less secure because the trust policy is generic.” Only if you leave it generic. Request tag conditions scope it as tightly as an IRSA trust policy, and you keep the session tags IRSA does not give you.
• “Run one mechanism per cluster, cleanly.” A mixed cluster is documented, supported, and for clusters with Fargate pods it is the permanent steady state, not a transitional mess.
• “EKS add-ons can only use IRSA.” True early on, and still repeated in older write-ups. Add-on creation and update now accept Pod Identity associations directly. Check your specific add-on rather than trusting a blog post, including this one.
• “Migrate everything, then delete the OIDC provider.” Deleting it breaks every role still trusting it, including ones you forgot. Migrate workload by workload, and delete the provider only after CloudTrail shows zero AssumeRoleWithWebIdentity calls for that issuer across a full batch cycle.

How I’d decide, and what I’d do first

• Default new workloads on EC2 or Karpenter-managed nodes to Pod Identity, and leave working IRSA alone until you have a reason to touch it.
• Add request tag conditions for anything sensitive. Portability helps a shared platform role and hurts a role that reads customer data.
• Restrict pod access to IMDS before you migrate anything, not after. It converts the silent failure into a visible one.
• Migrate with the role trusting both principals at once, so rollback is a manifest change rather than an IAM change.
• Put credential identity in your dashboards. A CloudTrail query counting calls made by node roles from workload namespaces catches the exact failure this post opened with. Amazon Managed Grafana or Grafana Cloud will render it if you already ship CloudTrail somewhere; a scheduled CLI check from a CI runner or a small VPS from Contabo or InterServer is a fine low-tech alternative.
• Codify associations in Terraform or OpenTofu from the start. They are cheap to create by hand and easy to lose track of.

Frequently asked questions

Is EKS Pod Identity replacing IRSA?

No. AWS recommends Pod Identity for new applications on supported node types and continues to support IRSA. Since Pod Identity cannot run on Fargate or Windows nodes, IRSA stays for the workloads that live there.

Can I use both in the same cluster?

Yes, and it is a documented, supported pattern. Older workloads on IRSA and new ones on Pod Identity is normal. Avoid configuring both for the same service account beyond a migration window, because which one wins depends on SDK credential chain ordering.

Why does my pod get AccessDenied after switching to Pod Identity?

Check the trust policy for sts:TagSession first. Pod Identity always sends session tags, so a trust policy allowing only sts:AssumeRole is rejected. After that, check that the namespace and service account in the association match the pod exactly.

Does Pod Identity work for cross-account access?

Yes, through role chaining. The association takes a role in the cluster account plus a target role in the resource account, and EKS chains them. It also exposes an external ID you can require in the target role’s trust policy to prevent a confused deputy problem.

Do I have to change my application code?

No, provided the SDK is recent enough. Both work through the default credential chain, so the application calls AWS as usual. The only change that bites is an SDK too old to understand the container credential provider.

How do I confirm which identity a pod is actually using?

Run aws sts get-caller-identity inside the container and read the assumed role ARN. If it names the node role rather than the one you assigned, the chain fell through. Cross-check in CloudTrail, where session tags also give you the namespace and service account for Pod Identity sessions.

The one thing worth remembering

The real question in EKS Pod Identity vs IRSA is not which is more modern. It is where you want the cluster-to-role relationship to live. IRSA puts it in IAM trust policies: precise, cluster-specific, and increasing administrative drag as clusters multiply. Pod Identity puts it in the EKS API: portable, fast to change, and it hands you the job of deciding per role how much of that portability you actually want.

Pick per workload, not per cluster. And whichever you pick, verify the identity from inside the pod, because the worst outcome here is not a permission error. It is a workload that has been running with the wrong credentials since the day you deployed it.

Need a second pair of eyes on your EKS workload identity?

Most of this work is unglamorous: reading trust policies, matching associations against manifests, proving what a pod is really authenticating as. Things I help with:

• Auditing every service account in a cluster and reporting which identity its pods actually use, not the one the manifest claims
• Planning and running an IRSA to Pod Identity migration workload by workload, with a rollback that needs no IAM change
• Writing scoped trust policies with request tag conditions, so portability is a deliberate choice per role
• Restricting pod access to IMDS and confirming nothing was silently depending on the node role
• Setting up cross-account access with target roles and external IDs, plus the CloudTrail view that proves it is used as intended
• Codifying associations and roles in Terraform or OpenTofu, so the next cluster is a variable change rather than a project

Send me a trust policy, a service account manifest, or the output of aws sts get-caller-identity from inside one of your pods, and I will tell you what it is really doing.

The post EKS Pod Identity vs IRSA: What Actually Decides the Choice appeared first on John Nessime.

The user pool was set to MFA: Required. Somebody had screenshotted that console page for the security questionnaire. The screenshot was accurate. It just didn’t mean what everyone assumed it meant.

That gap is the whole problem with Cognito MFA for healthcare apps. The service gives you a setting that reads like a guarantee, and then quietly carves out entire classes of users, sessions, and sign-in paths that the setting never touches. Nothing errors. Nothing logs a warning. Your compliance evidence looks fine right up until someone actually tests it.

This post covers what the MFA switch in Amazon Cognito actually controls, the five failure families that show up in real healthcare deployments, how to wire a user pool so the enforcement matches the claim, and what to check when a clinician calls the help desk because they can’t get in.

What HIPAA actually requires from MFA right now

Worth getting this straight before touching any configuration, because a lot of vendor content is confidently wrong about it.

The HIPAA Security Rule as it stands today does not name multi-factor authentication. It requires person or entity authentication, and it leaves the mechanism to your risk analysis. MFA is how most people satisfy that in practice, but the rule does not spell it out.

What changes the picture is the Notice of Proposed Rulemaking that HHS published in the Federal Register on January 6, 2025. It would make MFA an explicit requirement for access to systems handling ePHI, with narrow exceptions, and it would remove the “addressable” escape hatch that currently lets organizations document a control as infeasible. This one matters enough to name the date, because timelines have moved: the comment period closed in March 2025, the rule is still proposed, and the Unified Agenda has pushed anticipated final action out to 2027. That estimate is not binding and has already slipped once.

Practical read: build for the proposed rule now, but don’t tell a client the mandate is already law. It isn’t, and auditors know the difference.

On the AWS side, Amazon Cognito is on the HIPAA Eligible Services list and is covered by the AWS Business Associate Agreement. Eligibility is a starting line, not a finish line. You still have to accept the BAA in each AWS account that touches PHI, and you still have to configure the service correctly. Nothing about creating a user pool makes it compliant.

Failure family one: half your users never see the MFA prompt

This is the one that produced the pen test finding, and it’s the most common serious gap I run into.

Cognito user pool MFA applies to local users. Those are the accounts that live in the user pool directory and authenticate against it. For federated users, the ones arriving through SAML or OIDC from a hospital identity provider, Okta, Entra ID, or an in-house ADFS, Cognito hands the entire authentication decision to the upstream IdP. It does not add a second factor. It cannot. It never sees the password.

So a portal with patients as local users and staff federated from a hospital directory has exactly the split you’d fear: patients get MFA, clinicians don’t, unless the hospital IdP is enforcing it independently. And the hospital IdP is usually somebody else’s system, run by a team you don’t have a Slack channel with.

There’s a second version of this. The app client, not the user pool, controls which authentication flows are permitted. If a legacy mobile client is still allowed to use a flow you thought you’d retired, that path stays open regardless of what the pool-level MFA setting says.

What to do about it:

• Inventory every app client attached to the pool. Not the ones you remember creating. Run list-user-pool-clients and account for all of them.
• For each federated IdP, get written confirmation of what the upstream enforces. A trust relationship is not a control. Put the answer in your risk analysis where an auditor will find it.
• Treat federation as a boundary you inherit risk across, and say so out loud in design reviews. This is the single most useful sentence you can contribute to a healthcare identity project.

Failure family two: the recovery attribute collision

This one is genuinely subtle, and it doesn’t bite during development. It bites during rollout, at scale, to the people least able to tolerate it.

Cognito refuses to let a user’s MFA delivery method be the same channel as their account recovery method. The logic is sound: if a password reset code and a second factor both land in the same inbox, an attacker with that inbox has everything, and the second factor is decoration.

The consequence is where it gets sharp. If MFA is required, a user has only an email address and no phone number, email is your account recovery channel, and they have not registered a TOTP authenticator, then that user cannot complete sign-in at all. Cognito returns an error. They are not locked out temporarily. There is no path forward from the login screen.

Now think about who that describes in a healthcare deployment. Part-time nursing staff enrolled from an HR export that had work email but no mobile number. Older patients who gave an email at registration and nothing else. Contractors. Locums. It is never a small number, and they all arrive at the help desk in the same week.

The same rule cuts the other way on password reset. A user whose preferred MFA is email cannot receive a reset code by email. If they have no phone number, self-service recovery is gone, and someone with administrative credentials has to intervene for every single one.

Three ways out, in the order I’d reach for them:

• Make TOTP the default factor. An authenticator app doesn’t consume either the email or phone attribute, so the collision never occurs. This is the one I reach for first in staff-facing apps.
• Require both email and phone_number as sign-up attributes. When a user has both, Cognito automatically routes reset codes to whichever channel isn’t their MFA factor. Clean, but it forces a phone number on people who may not want to give one.
• Configure a recovery fallback. Set account recovery to prefer email but fall back to SMS, so users who aren’t eligible for email recovery still have a route.

Whatever you choose, run a query against your existing user records for accounts missing a phone number before you flip MFA to required. Finding those users in a report is cheap. Finding them through a help desk queue is not.

Failure family three: required MFA and adaptive authentication are mutually exclusive

This trips up almost every security-conscious design I review, because both features are things a healthcare client will ask for by name.

Cognito’s threat protection includes adaptive authentication: risk scoring on each sign-in, with automated responses like requiring MFA or blocking outright. To use those automated responses in full-function mode, MFA must be set to optional in the user pool. That is a documented constraint, not a bug. The mechanism only makes sense if there’s a state where MFA is not already being demanded.

So you cannot check both boxes. Either MFA is required for everyone and adaptive responses have nothing to escalate to, or MFA is optional and risk scoring can escalate, which means low-risk sign-ins may proceed on a password alone.

For anything touching ePHI, I’d take required MFA and give up the adaptive escalation. A control that always applies beats a control that usually applies, and “usually” is a hard thing to defend in an audit.

You don’t have to give up the visibility, though. Threat protection runs in audit-only mode alongside required MFA. You still get risk signals, device fingerprinting, and compromised credential detection surfaced in CloudWatch and in the user event logs, without the enforcement conflict. That’s a genuinely good trade: enforcement from the required setting, detection from threat protection.

Two things to know before you budget for it. Threat protection lives in the Plus feature tier, and audit-only mode is billed the same as full-function. There’s no free observation period. Export those events somewhere you’ll actually look at them, whether that’s a CloudWatch Logs destination feeding your SIEM, or a platform like Datadog or Grafana Cloud if that’s where your on-call already lives.

Failure family four: the session outlives the shift

MFA protects the moment of sign-in. It does nothing about the eight hours after it. In healthcare that gap is unusually expensive, because the workstation is shared, the browser stays open, and the next person to touch the keyboard is a different human being.

Cognito’s defaults are built for consumer apps. Access and ID tokens last one hour. Refresh tokens last thirty days. Neither default is wrong in general and both are wrong for a clinical workstation.

The ranges you have to work with:

• Access token and ID token: five minutes to one day, set per app client.
• Refresh token: sixty minutes to ten years, set per app client. It cannot be shorter than the access token lifetime.

Now the part that catches people, and the reason I put this failure family on the list. If you use managed login, Cognito sets a browser session cookie valid for one hour. Setting your access token to fifteen minutes does not shorten the re-authentication window. During that hour the user’s session refreshes without presenting credentials again, because the cookie is still good. Teams configure short tokens, write “15-minute session timeout” in the security documentation, and the observable behavior is an hour.

The second trap is revocation. Revoking a token, or a global sign-out, invalidates refresh tokens. It does not invalidate an access token that has already been issued. That token stays valid until it expires, and any API accepting it will keep accepting it. So the practical blast radius of a compromised session is your access token TTL, not the moment you clicked “sign out this user.”

That’s the argument for short access tokens: it is the only lever that shortens the window between “we revoked access” and “access actually stopped.”

# Tighten an app client used by clinical staff.
# Access tokens live 15 minutes; refresh tokens expire at the end of a shift.
aws cognito-idp update-user-pool-client 
  --user-pool-id us-east-1_EXAMPLE 
  --client-id EXAMPLECLIENTID 
  --access-token-validity 15 
  --id-token-validity 15 
  --refresh-token-validity 9 
  --token-validity-units AccessToken=minutes,IdToken=minutes,RefreshToken=hours 
  --enable-token-revocation

Refresh token rotation is worth turning on where your client can support it. Each refresh returns a new refresh token and invalidates the old one, so a stolen refresh token is usable once at most. Two caveats that bite in practice: rotation does not extend the original lifetime, so a nine-hour refresh token still dies nine hours after sign-in no matter how often it rotated, and app clients with rotation enabled cannot use the older REFRESH_TOKEN_AUTH flow. That second one has broken more than one mobile release. Rotation requires the Essentials tier or higher.

None of this replaces workstation-level controls. Screen lock policy, OS session timeouts, and endpoint hardening tooling like the utilities from O&O Software do work your identity layer cannot do from the outside. If staff reach the admin console remotely, a managed VPN such as NordLayer or Surfshark reduces network exposure, but be honest in your documentation about what that is. It is not an authentication control, and treating it as one is exactly the reasoning auditors push back on.

Failure family five: the factor you picked is phishable

Cognito offers SMS, email, and TOTP as second factors, plus passkeys as a first factor under specific conditions. They are not equivalent, and the console presents them as though they were.

SMS is the weakest. SIM swap and SS7 interception are real, and an SMS code is trivially relayed through a phishing proxy. It has one genuine advantage worth naming: enrollment friction is near zero, and for elderly or low-technology-comfort patient populations that is not a small thing. An SMS factor that gets used beats a TOTP factor that gets abandoned. Just don’t put clinical staff behind it. There’s also an operational trap: if your account is in the Amazon SNS SMS sandbox in the relevant region, you must verify each destination number before delivery works, which turns a smooth rollout into a stalled one.

Email OTP is roughly SMS-equivalent in phishing resistance and worse if the mailbox is on the same laptop as the session you’re protecting. It needs the Essentials tier or higher and a configured Amazon SES sending identity. Useful as a fallback, weak as a primary.

TOTP is the sensible default for staff. It’s phishable, the code can be relayed in real time by an attacker-in-the-middle, but it’s immune to SIM swap and it doesn’t collide with your account recovery channel. Any standard authenticator works. If your organization already uses a password manager with built-in TOTP, such as 1Password or Bitwarden, enrollment gets meaningfully easier, at the cost of both factors living in one vault. That’s a trade-off to make deliberately rather than by accident.

Passkeys are the only phishing-resistant option here, and their integration has a shape you need to understand. A passkey cannot be a second factor on top of a password in Cognito. It can satisfy the MFA requirement on its own, as a first factor, when you set FactorConfiguration to MULTI_FACTOR_WITH_USER_VERIFICATION in the pool’s WebAuthn configuration. The user verification step, biometric or device PIN, is what makes it count as multi-factor.

That’s the path I’d push for privileged accounts: administrators, anyone with bulk record export, anyone who can change the pool configuration itself. Platform authenticators cover most staff; hardware keys like YubiKey are worth the cost for the handful of accounts whose compromise ends the conversation.

One hard constraint to plan around: you cannot set MFA to required in a pool that allows email or SMS one-time passwords as a first factor. Passwordless OTP sign-in and required MFA are incompatible. Passkeys with user verification are the documented exception.

Wiring up Cognito MFA for healthcare apps

An order of operations that avoids locking people out. Do this in a non-production pool first, with synthetic users only.

1. Accept the BAA in the AWS account through AWS Artifact, before any real user data lands in the pool. The BAA is per-account and is not inherited by new accounts.
2. Audit your user records for accounts missing phone_number or email. This is the population that will fail after you enable required MFA.
3. Decide the recovery configuration and set it explicitly, with a fallback channel, so the MFA factor and the recovery channel can never be the same for any user.
4. Enable MFA methods with the pool set to optional first. Let users enroll voluntarily. Track enrollment coverage as a number you report weekly.
5. Flip to required only when coverage is high and the stragglers have a support path. Managed login will prompt unenrolled users to set up a factor at next sign-in, but only when MFA is required.
6. Tighten token lifetimes per app client, then verify the observable behavior rather than the configuration. If you use managed login, remember the one-hour cookie.
7. Turn on threat protection in audit-only mode and stream the events out to wherever you review security signal.

The core enforcement setting, once you’re ready:

# Require MFA pool-wide, TOTP as the available method.
aws cognito-idp set-user-pool-mfa-config 
  --user-pool-id us-east-1_EXAMPLE 
  --mfa-configuration ON 
  --software-token-mfa-configuration Enabled=true

# Set a specific user's preferred factor administratively.
aws cognito-idp admin-set-user-mfa-preference 
  --user-pool-id us-east-1_EXAMPLE 
  --username clinician@example.org 
  --software-token-mfa-settings Enabled=true,PreferredMfa=true

Note that admin-set-user-mfa-preference does not reset an existing TOTP registration. To issue a new authenticator secret for a user who lost their phone, you go through AssociateSoftwareToken and VerifySoftwareToken. Build that runbook before you need it, because you will need it, and improvising an identity-proofing procedure at 2pm on a Tuesday is how bad help desk habits start.

A staging pool holding only synthetic patients doesn’t need to live inside the BAA account, and there’s no reason a test environment should carry production’s cost structure. A plain VPS from InterServer or Contabo is fine for the app tier in front of a throwaway pool, provided you are certain no real PHI can reach it. That certainty is the whole condition, and it’s worth writing down rather than assuming.

Troubleshooting

A user gets an error at sign-in and never reaches an MFA prompt

Almost always the recovery collision. Check whether the account has both email and phone_number, and compare the populated attribute against the pool’s account recovery setting. If the only available attribute is also the recovery channel, and there’s no TOTP registration, the user has no eligible factor. Register TOTP for them or add the missing attribute.

Sign-in succeeds without MFA even though the pool says required

Work backward from the user, not from the pool. Is this a federated user? Then Cognito is not in the authentication path and pool MFA does not apply. Is it their first sign-in after account creation? Cognito treats the initial verification message as the second factor on that first sign-in and prompts for factor registration afterward. Is a different app client involved with different permitted flows?

Password reset stopped working after enabling MFA

Expected behavior, not a regression. The reset code cannot go to the same channel as the user’s MFA factor. Users with only one contact attribute lose self-service recovery. Add the second attribute or configure a fallback recovery mechanism.

SMS codes never arrive

Check the Amazon SNS SMS sandbox status in the region holding your SNS resources. In the sandbox, destination numbers must be verified in SNS before delivery. Also confirm the IAM role Cognito assumes for SNS is present and correctly scoped.

Repeated MFA failures lock a user out

After five unsuccessful MFA code attempts, Cognito starts an exponential timeout lockout. It resolves on its own. Most real cases are clock drift on the user’s phone breaking TOTP validation, so check that before assuming an attack.

Tokens keep working after you revoked them

Correct and by design. Revocation kills refresh tokens; already-issued access tokens survive to their expiry. If that window is unacceptable for your risk profile, shorten the access token TTL. There is no other lever short of checking revocation on every API call.

Common mistakes

• Treating the pool-level MFA setting as coverage for federated users. It is not, and this is the gap most likely to survive to production.
• Storing PHI in Cognito custom attributes. Diagnosis codes, MRNs, and care team assignments do not belong in a directory record that gets copied into ID token claims, which then land in browser storage, proxy logs, and error reports. Keep the identity layer to identity.
• Enabling required MFA in one deploy without an enrollment ramp. You will manufacture a support incident.
• Assuming a short access token gives a short session when managed login is in play. The one-hour cookie governs re-authentication.
• Believing revocation is immediate.
• Leaving threat protection off because required MFA is on. Audit mode and required MFA coexist, and the detection signal is separately valuable.
• Enabling refresh token rotation without checking whether your client uses REFRESH_TOKEN_AUTH. That combination is not permitted and it breaks at runtime.

Best practices

• Separate app clients for patient-facing and staff-facing applications, with independently tuned token lifetimes. Staff sessions should not inherit consumer defaults.
• TOTP as the baseline factor for anyone with access to more than their own record.
• Passkeys with user verification for administrators and bulk-export roles.
• Both email and phone_number collected at enrollment where the population allows it, purely to keep recovery working.
• Access token TTL short enough that revocation means something. Fifteen minutes is a reasonable starting point for clinical apps.
• Threat protection in audit mode, with events exported and actually reviewed.
• Manage the pool in Terraform or OpenTofu so MFA settings, recovery configuration, and token lifetimes are reviewable in a diff instead of discovered in the console during an audit.
• Test the MFA path in CI against a non-production pool. Configuration drift in an identity provider is silent by nature.

FAQ

Is Amazon Cognito HIPAA compliant?

No service is HIPAA compliant by itself. Cognito is HIPAA eligible and covered under the AWS Business Associate Agreement, which means you may process PHI with it once you have accepted the BAA in that account and configured it appropriately. Compliance is a property of your whole system and your documentation, not of the service.

Does HIPAA require MFA today?

Not explicitly. The current Security Rule requires person or entity authentication and leaves the method to your risk analysis. The proposed 2025 update would make MFA an express requirement, but it remains proposed and the timeline for final action has moved more than once.

Can I use required MFA and adaptive authentication together?

Not in full-function mode. Adaptive authentication’s automated responses require MFA to be optional in the user pool. You can run threat protection in audit-only mode alongside required MFA, which keeps the detection signal without the conflict.

Which MFA factor should healthcare apps use?

TOTP as the baseline for staff, passkeys with user verification for privileged accounts. SMS is defensible for patient populations where enrollment friction genuinely determines whether MFA gets used at all, but it should not protect clinical access.

Does Cognito MFA apply to users signing in through our hospital SSO?

No. For federated users arriving via SAML or OIDC, Cognito delegates authentication entirely to the upstream identity provider and does not add a second factor. Enforcement has to happen at the IdP, and you should get that in writing.

How short should session timeouts be for a clinical application?

Set access tokens short, commonly around fifteen minutes, and refresh tokens to roughly a shift length rather than the thirty-day default. Then verify what users actually experience, because managed login’s one-hour browser cookie can keep a session alive past your token settings.

Can I store patient identifiers in Cognito user attributes?

You can, and I would avoid it. Custom attributes flow into ID token claims, which end up in browser storage, application logs, and error tracking. Keep an opaque internal identifier in the token and resolve clinical data from your application database behind its own authorization check.

Conclusion

If you remember one thing about Cognito MFA for healthcare apps, make it this: the MFA setting is a claim about local users at the moment of sign-in, and nothing else. It says nothing about your federated clinicians, nothing about the session that continues for the next eight hours, and nothing about whether the factor you chose survives a phishing proxy.

Verify enforcement by observing behavior, per user population, per app client. Not by screenshotting the console page. That screenshot is what got flagged in the pen test at the top of this post, and it was completely accurate.

Need help with Cognito and healthcare authentication?

I work with teams building and hardening authentication on AWS, particularly where a compliance obligation makes “it works” insufficient. Things I can help with:

• Auditing an existing user pool for the gaps in this post: federated coverage, recovery collisions, token lifetimes, app client sprawl
• Designing an MFA rollout that won’t generate a help desk queue, including the pre-flight query against your user records and the enrollment ramp
• Implementing passkeys with user verification for privileged accounts, alongside TOTP for general staff
• Moving user pool configuration into Terraform or OpenTofu so identity settings are reviewable and drift is visible
• Wiring threat protection events into your existing monitoring and building the alerts that are worth waking someone for
• Writing the technical evidence that maps your configuration to your risk analysis, in language an auditor accepts

If you have a user pool that isn’t behaving the way the console suggests it should, send me the output of describe-user-pool and describe-user-pool-client with the identifiers stripped, and I’ll tell you what I see.

The post Cognito MFA for Healthcare Apps: The Settings That Look Compliant but Aren’t appeared first on John Nessime.

The integration timeout had been sitting in front of that API since the day someone created it. It only became a bug when a customer’s dataset got big enough to cross it.

That is the real shape of the ALB vs API Gateway decision. You’re not picking a feature list. You’re picking a set of hard limits, a billing curve and an authentication story, then living inside them. The failure mode that bites isn’t a bad Friday, it’s finding the constraint months later, once the thing in front of your app has become load-bearing.

Below: how each one bills you, which limits surface late, what changed recently on the load balancer side, and a decision procedure.

The old rule of thumb is shakier than it looks

The advice you find everywhere is: containers and instances behind an Application Load Balancer, serverless behind API Gateway. Wrong often enough to cost you. ALB takes Lambda functions as targets. API Gateway reaches containers through private integrations with an NLB, an ALB, or Cloud Map. Compute type barely narrows the field.

What separates them is who’s calling and what needs enforcing before a request reaches your code. A browser session against your own web app is a different problem from a partner’s server calling a documented endpoint with a key you issued and a quota you promised. One needs routing and a session. The other needs a contract.

Application Load Balancer: routing you own the position of

ALB is a Layer 7 router inside your VPC. You give it subnets and security groups, attach listeners, and write rules that match on host, path, header, query string, method or source IP, then forward, redirect, return a fixed response, or authenticate.

Where it wins

• Long requests. The idle timeout is an attribute you set, defaulting to 60 seconds. Report generation and slow third-party calls become a number in Terraform instead of an architecture problem.
• Big bodies and streaming. Against instance, IP and container targets there’s no small payload ceiling to design around, and WebSocket upgrades pass through.
• Browser authentication. An HTTPS listener rule can run an authenticate-oidc or authenticate-cognito action before forwarding. The load balancer does the redirect dance, sets a session cookie, and hands your target the verified claims in a header. Your app reads a header instead of implementing OAuth.

Where it doesn’t

• No per-client rate limiting. No API keys with quotas attached. Rate limiting means AWS WAF rate-based rules in front, counting requests per source over a window. Blunter than a per-consumer plan, and a separate bill.
• No request validation or response caching. A malformed body reaches your handler and burns your compute. A cacheable GET is recomputed unless you put a CDN in front.
• Lambda targets are constrained. Request body capped at 1 MB, response JSON capped at 1 MB, WebSocket upgrades rejected with a 400. If your function returns a large document, that ceiling arrives as a 502 in production, not an error at deploy time.
• You pay for it while it sleeps. An hourly charge per load balancer whether or not a request arrives. Three environments, three floors.

Amazon API Gateway: the contract layer

API Gateway isn’t a load balancer with extra features. It’s a place to define what your API promises: which routes exist, who may call them, how often, and in what shape. That framing explains most of its trade-offs.

Choosing it is really two decisions, because REST APIs and HTTP APIs are different products under one name.

• REST APIs carry the full management surface: API keys, usage plans, per-client throttling, request validation, response caching, AWS WAF integration, the private endpoint type, execution logs and X-Ray tracing.
• HTTP APIs are the stripped-down, cheaper option. They add a built-in JWT authorizer, automatic deployments and Cloud Map private integrations. They offer none of the six management features listed above.

That second line is the one people get wrong. HTTP APIs aren’t “REST APIs but cheaper”, they’re a different product that happens to serve HTTP. Pick them to save money, then discover you need metered access, and you’re rebuilding usage plans in your own code.

Where it wins

• You’re selling or governing access. API keys tied to usage plans, on REST APIs, are the only native answer to “customer B must not burn customer A’s capacity”.
• Rejecting bad requests before they cost you. Request validation against a model means a malformed payload never invokes your function. On per-invocation compute that’s a security boundary and a cost lever at once.
• Zero idle cost. No hourly floor, so a dev stage nobody calls is free. For genuinely spiky traffic this is often the whole argument, and there’s no network to design either.

Where it doesn’t

• The integration timeout. The default is 29 seconds. It can be raised, but AWS notes that going beyond 29 seconds may require reducing your Region-level throttle quota. You’re buying request duration with request rate.
• Payload ceilings. There’s a documented maximum payload size, well below what a file upload service wants. Large objects want presigned S3 URLs either way, but here the gateway forces that conversation early.
• Cost scales with request count. Per request plus data transfer. A health check hammered by a monitoring tool shows up directly on the line item.
• The account throttle is shared. The default steady-state rate and burst quota applies per Region across your APIs in that account. Raisable, but a shared ceiling by default rather than a per-API one.

What changed: the load balancer validates JWTs itself now

For years the clean argument for API Gateway on machine-to-machine APIs was token validation. If a partner’s service calls you with a bearer token, someone has to check signature, issuer and expiry, and you’d rather that not be a library duplicated in every service.

ALB does this natively now. A rule on an HTTPS listener can carry a jwt-validation action that runs before any routing action. The load balancer fetches signing keys from a JWKS endpoint you configure, checks the signature, and requires iss and exp. It also validates nbf and iat when present, and you can configure up to ten additional claims. Valid tokens forward unchanged. Invalid ones never reach your VPC.

This isn’t authenticate-oidc, which is built for humans in browsers and works by redirecting. JWT validation does no redirect. It inspects the token and returns a yes or no, which is what service-to-service traffic needs. Validation action first, routing action second, and order matters:

aws elbv2 create-rule 
  --listener-arn "$LISTENER_ARN" 
  --priority 10 
  --conditions Field=path-pattern,Values="/api/*" 
  --actions '[
    {
      "Type": "jwt-validation",
      "JwtValidationConfig": {
        "JwksEndpoint": "https://issuer.example.com/.well-known/jwks.json",
        "Issuer": "https://issuer.example.com"
      },
      "Order": 1
    },
    {
      "Type": "forward",
      "TargetGroupArn": "$TARGET_GROUP_ARN",
      "Order": 2
    }
  ]'

Two constraints first. Only RS256 is supported, and the JWKS endpoint has a maximum response size and key count. A provider publishing a large key set fails that check, and the failure mode is that requests stop reaching your targets. Fetch your provider’s JWKS document and look at it before committing.

So “we need token validation at the edge” is no longer by itself a reason to reach for API Gateway. What the gateway still owns is everything downstream of that yes or no: keys, quotas, per-consumer throttles, shape validation, caching.

How each one actually bills you

This is where the argument usually goes wrong, because people compare unit prices instead of curve shapes.

API Gateway is a slope with no floor: per request, plus data out. Zero traffic costs nothing, ten times the traffic costs roughly ten times as much.

ALB is a floor plus a stranger slope. An hourly rate per load balancer, plus capacity units measured across four dimensions at once:

• New connections per second
• Active connections per minute
• Processed bytes per hour
• Rule evaluations per second, where the first ten processed rules per request are free

You’re charged on the highest of the four in a given hour, not the sum. That mechanic explains most surprises. Clients that open a fresh connection per call can bill you on new connections while your bandwidth graph looks flat. A listener with dozens of path rules can bill you on rule evaluations while everything else idles.

Two details worth knowing: Lambda targets get a smaller processed-bytes allowance per capacity unit than instance, container and IP targets, so identical traffic consumes more units. Enabling mutual TLS reduces the active-connections allowance per unit.

Before modelling anything, look at what you already consume:

aws cloudwatch get-metric-statistics 
  --namespace AWS/ApplicationELB 
  --metric-name ConsumedLCUs 
  --dimensions Name=LoadBalancer,Value=app/my-alb/0123456789abcdef 
  --start-time "$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)" 
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" 
  --period 3600 
  --statistics Sum Maximum

Run that against a representative week and you have the real input. For API Gateway, the equivalent is request count per stage. Across several accounts, a cost visibility tool like Vantage or CloudZero attributes this per environment faster than Cost Explorer, though disciplined tagging gets you most of the way for free.

Rough shape: low steady traffic favours API Gateway because you skip the floor. High-volume, byte-heavy, long-lived-connection traffic favours ALB because the per-request slope disappears. Chatty traffic with tiny payloads is where the curves cross, and that one is worth calculating.

A decision procedure for ALB vs API Gateway

Run these in order, stop at the first clear answer.

1. Does any request legitimately run past about thirty seconds? If yes and you can’t make it asynchronous, ALB. Raising the gateway timeout is possible but trades against your regional throttle quota.
2. Do you need per-consumer quotas or metered access? If yes, API Gateway REST APIs. Nothing else here gives you keys and usage plans natively.
3. Do requests or responses carry large payloads? If yes, ALB with instance or container targets. Both options constrain size, but the Lambda-target and gateway ceilings arrive sooner.
4. Is traffic occasional, spiky, or mostly zero? If yes, API Gateway. An hourly floor for a few thousand requests a day is real cost with no matching benefit.
5. Is the caller a browser session against your own app? If yes, ALB, with an authenticate action if you need login. The gateway’s management features are dead weight here.
6. Is the caller another service with a bearer token and nothing more? Either works now. Decide on cost curve and payload shape, because JWT validation stopped being the tiebreaker.

Still torn? Choose the limits you’d rather explain to a customer in a year. Usually ALB for applications, API Gateway for products. Whichever you pick, alarm on the constraint rather than the symptom: integration latency approaching the timeout, or consumed capacity units trending up. Grafana Cloud or plain CloudWatch alarms both do the job. Then send a payload just over your ceiling in staging and confirm what the client actually receives.

Arguments that don’t survive contact

“It’s serverless, so it’s cheaper”

Per-request billing is cheaper at low volume and more expensive at high volume. Arithmetic, not philosophy. The crossover depends on your request size and connection reuse, which is why the capacity-unit metric matters more than anyone’s price table.

“We’ll put API Gateway in front of it later”

You can, through a private integration, and it’s a legitimate pattern. But “later” means a second hop, a second bill, a second set of timeouts and a second place to look during an incident. Adding it deliberately is fine. Assuming it’s free is not.

“ALB has no rate limiting, so it’s insecure”

ALB has no native rate limiting, but WAF rate-based rules in front of it are a real control, and a source-based limit is often closer to what you wanted than a per-key quota. Cloudflare in front of the load balancer is another common answer when you already want its caching and bot controls. What ALB genuinely lacks is the per-consumer contract, which matters when consumers are customers and doesn’t when they’re your own front end.

Check the pairing before you rely on it, though: WAF associates with ALB and with API Gateway REST APIs, but not with HTTP APIs. And the amount of request body WAF reads is configurable for some resource types and fixed for the load balancer. Anything past that window isn’t scanned.

“The front door has to be an AWS service”

For one service with predictable traffic, a plain VPS running Nginx or Caddy does the same job with far less surface. Providers like Contabo or InterServer are reasonable homes for a small internal API. It won’t scale into a multi-service platform and it moves patching and certificate renewal onto you, but “which AWS service” is sometimes the wrong question.

Frequently asked questions

Can I use API Gateway and ALB together?

Yes. API Gateway supports private integrations with an Application Load Balancer, so the gateway handles keys, quotas and validation while the load balancer routes inside the VPC. Sound pattern for a public product API in front of an existing internal platform. It also doubles the hops, the bills and the places a request can die, so use it when you need both jobs done.

Which is cheaper, ALB or API Gateway?

Neither, universally. API Gateway has no idle cost and charges per request, so it wins at low or intermittent volume. ALB charges an hourly floor plus capacity units billed on the highest of four dimensions, so it wins once the per-request slope dominates. Pull your consumed capacity units and request counts and do the arithmetic on your own traffic.

Can ALB do rate limiting?

Not natively. You get it by associating AWS WAF with the load balancer and using rate-based rules, which limit by source over a time window. If you need per-key quotas, that’s an API Gateway REST API usage plan, or logic you write yourself.

Does ALB support JWT authentication now?

Yes, through a jwt-validation action on an HTTPS listener rule. It verifies the signature against a JWKS endpoint and checks issuer and expiry, plus optional additional claims. Note the RS256-only restriction and the JWKS size and key count limits before designing around it.

Should I use HTTP APIs or REST APIs?

REST APIs if you need any of: API keys, usage plans, per-client throttling, request validation, response caching, WAF integration, or a private endpoint type. HTTP APIs when you need none of those and want the built-in JWT authorizer at a lower price. Migrating between them isn’t a config flag, so decide before you publish a URL.

The one thing worth remembering

ALB vs API Gateway isn’t a question about serverless versus containers, and it stopped being a question about token validation. It’s a question about whether the thing in front of your application needs to enforce a contract with people outside your organisation.

If it does, take the gateway and accept its timeout and payload ceilings as the price of keys, quotas and validation. If it doesn’t, take the load balancer and spend the saved effort on the network and the health checks. Either way, write down the three numbers that will eventually bite: timeout, payload ceiling, throttle quota. The 504 in that opening ticket wasn’t a bug. It was a decision nobody had written down.

Need a second opinion on your front door?

Cheap to get right at the start, expensive to revisit. If you’re staring at this now, or already living with the wrong answer:

• Modelling ALB capacity units against API Gateway request counts on your actual traffic, so the comparison uses your numbers
• Auditing an existing gateway or load balancer for the limits you’re closest to hitting: timeouts, payload ceilings, throttle quotas
• Moving browser authentication off your application onto an ALB authenticate rule, or setting up JWT validation at the listener for service-to-service traffic
• Designing the gateway plus private integration pattern when you need both layers, including where to terminate TLS and how to preserve client IPs
• Building the Terraform or OpenTofu modules for either option, with listener rules, target groups, health checks and alarms in one place
• Setting up alarms that fire on approaching limits rather than the 5XX that follows

Send me a listener rule dump, a stage configuration, or a week of consumed capacity unit metrics and I’ll tell you what I see.

The post ALB vs API Gateway: Pick the Limits You Can Live With appeared first on John Nessime.

Here is the thing that catches teams out: the pipeline definition is the cheapest part of a CI system to move. Rewriting stages and steps is a few days of tedious work for a normal service. What does not move is everything underneath it, and that is where a two-week migration turns into a two-quarter one.

This post compares GitLab CI vs GitHub Actions vs Jenkins on the things that actually differ once you are running them in anger: who owns the runners, where secrets live, how the bill is calculated, and what breaks when an outside contributor opens a pull request. There is a profile of each with an honest “where it wins and where it doesn’t”, a section on cost mechanics, a section on the security failure modes, and a decision procedure at the end you can run in an afternoon.

The part of the decision that actually bites

Draw a line under your pipeline file. Everything above the line is syntax. Everything below is platform.

Below the line sits the runner fleet and how it is scaled, the secret store and how jobs authenticate to it, artifact and container registry storage with its retention rules, deploy credentials and the network path they travel, approval gates and who can click them, and the audit trail that someone in compliance will eventually ask about. None of that ports. All of it has to be rebuilt on the new platform, and each piece has a person or a team attached to it who has opinions.

The failure mode this produces is specific and predictable. A team migrates the build and test half of the pipeline because that part is easy and demos well. Deploy stays on the old system “for now”, because deploy touches production and nobody wants to be the one who broke it during a migration. Six months later there are two CI systems, two secret stores, two sets of runners to patch, two audit trails, and a deploy path that nobody can explain end to end. The migration never technically failed. It just never finished, and the running cost doubled permanently.

So the real question is not which YAML dialect you prefer. It is whether you are prepared to move the whole thing, and which platform makes the “below the line” half cheapest for your specific situation.

What each platform is really optimising for

Feature tables make these three look more similar than they are. They are optimising for genuinely different things, and the differences show up as soon as you leave the happy path.

GitHub Actions: shortest path from push to green

Actions is built around events in the GitHub repository. Something happens, a workflow fires, jobs land on hosted runners, and you did not have to provision anything. The marketplace means most common tasks are already someone else’s problem. Reusable workflows let a platform team define a pipeline once and have every service call it.

A minimal hardened job header looks like this, and every line in it is there for a reason:

name: ci

on:
  pull_request:
    branches: [main]

permissions:
  contents: read      # start read-only, raise per job where needed
  id-token: write     # only for jobs that federate to a cloud provider

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@PIN_TO_A_40_CHAR_COMMIT_SHA
      - name: Use the PR title safely
        env:
          PR_TITLE: ${{ github.event.pull_request.title }}
        run: echo "$PR_TITLE"

The permissions block caps what the automatically provided job token can do, so a compromised step cannot push to your branches. The concurrency block cancels superseded runs when someone pushes three times in a minute, which is the single easiest compute saving available. Pinning the action to a commit SHA rather than a tag matters because tags are mutable and have been repointed to malicious commits in real supply chain incidents. And binding the pull request title to an environment variable instead of interpolating it directly into the shell is what stops an attacker-controlled string from being executed as a command.

Where it wins: your code is already on GitHub, you want CI running today, and your deploy targets are cloud APIs you can reach over the internet with a federated identity token. For open source it is close to unbeatable, since standard hosted runners are free for public repositories.

Where it doesn’t: anything needing deep control of the execution environment, long-running or stateful builds, deploys into networks the internet cannot reach, or a compliance posture that requires build logs and secrets to stay inside your perimeter. Self-hosted runners solve the network problem but hand you back the maintenance you were trying to avoid.

GitLab CI: one platform owning the whole lifecycle

GitLab’s pitch is that source control, CI, package registry, container registry, security scanning and issue tracking are one product with one permission model and one audit trail. That is a real architectural advantage, not marketing. When a scanner finding, the merge request that introduced it, the pipeline that caught it and the registry image that contains it are all the same system, you spend a lot less time gluing tools together.

The pipeline language rewards you for using its graph features rather than treating stages as a queue:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

default:
  interruptible: true

stages: [build, test, deploy]

build:
  stage: build
  script:
    - make build
  artifacts:
    paths: [dist/]
    expire_in: 1 week

unit:
  stage: test
  needs: [build]
  script:
    - make test

deploy:
  stage: deploy
  needs: [unit]
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  script:
    - ./deploy.sh

The workflow:rules block is the one people skip and then wonder why every push runs two pipelines. It decides whether a pipeline is created at all. needs turns the stage list into a directed graph, so a job starts the moment its own dependency finishes rather than waiting for every job in the previous stage. interruptible: true makes jobs eligible for automatic cancellation when a newer pipeline supersedes them, which you then have to enable in the project’s CI settings; the keyword alone does nothing. And expire_in is worth setting deliberately, because artifact storage is billed separately from compute and quietly accumulates.

Where it wins: you want one vendor for the whole lifecycle, you have compliance requirements that benefit from a single auth model and audit trail, or you need self-managed hosting without giving up the integrated experience. The self-managed edition with your own runners is genuinely one of the better value propositions in this space at moderate to high build volume.

Where it doesn’t: your code is not in GitLab. The CI is tightly coupled to GitLab repositories, and running it against code hosted elsewhere is possible but perpetually awkward. The third-party ecosystem is also smaller, so more of what you need you will write yourself. And self-managing GitLab is a real operational commitment, not a Docker Compose file you forget about.

Jenkins: whatever you need, on hardware you control

Jenkins gets dismissed as legacy by people who have never had a requirement it was the only tool to meet. It runs on your hardware, in your network, with no dependency on a vendor’s availability or pricing decisions. It talks to source control systems the other two do not. It can drive hardware test rigs, mainframe jobs, and build agents on operating systems nobody else offers. If your pipeline data legally cannot leave your network, this is a short list and Jenkins is on it.

pipeline {
  agent { label 'linux-ephemeral' }

  options {
    timeout(time: 30, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '30'))
    timestamps()
  }

  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
    stage('Deploy') {
      when { branch 'main' }
      steps {
        withCredentials([string(credentialsId: 'deploy-token', variable: 'DEPLOY_TOKEN')]) {
          sh './deploy.sh'
        }
      }
    }
  }

  post {
    always {
      cleanWs()
    }
  }
}

Two things in that file are worth pointing at. timeout and buildDiscarder are the difference between a controller that runs for years and one that fills its disk and hangs on stuck jobs; set them on every pipeline, not as an afterthought. And timestamps() and cleanWs() both come from plugins rather than core. That is Jenkins in a sentence: the capability exists, and it arrives as a dependency you now own, patch, and eventually discover is unmaintained.

Where it wins: air-gapped or heavily restricted networks, source control that is not GitHub or GitLab, orchestration of things that are not code builds, and organisations with genuinely unusual workflows that no hosted product models. If you already have working Jenkins infrastructure and a person who understands it, the case for ripping it out is weaker than the migration slides suggest.

Where it doesn’t: the controller is a stateful, single-point-of-failure server that someone has to own. Plugins are the extensibility model and also the attack surface, and the project publishes security advisories regularly enough that patching is a standing chore rather than an event. The Java baseline also moves; recent LTS lines have dropped support for older Java versions, so an upgrade is periodically a JVM migration too. If you cannot name the person responsible for that work, do not choose Jenkins.

GitLab CI vs GitHub Actions vs Jenkins on cost

Published rates change often enough that quoting them ages a post badly. The billing mechanics change far more slowly, and the mechanics are what you actually need to model.

• GitHub Actions. Standard hosted runners are free for public repositories. Private repositories draw from a monthly pool of included minutes tied to your plan, and those minutes are Linux-equivalent: Windows and macOS jobs drain the pool at a multiplier, with macOS by far the steepest. Larger runners do not draw from the included pool at all and bill per minute from the first second. Billing is per job rounded up to the minute, minutes do not roll over, and usage is charged to the repository owner. Self-hosted runner usage carries no per-minute fee, though GitHub has floated changing that and then walked it back, so verify against current billing docs before you build a model on it.
• GitLab CI. Jobs on GitLab-hosted instance runners consume compute minutes, calculated as job duration multiplied by a cost factor for the machine size. The included pool is per top-level namespace, not per seat, which is the detail that surprises people: a five-person group and a fifty-person group on the same tier start from the same pool. Self-managed runners consume zero quota on every tier including Free, which is why “pay for seats, run your own runners” is such a common GitLab shape.
• Jenkins. No licence cost, which is where most cost comparisons stop and where the interesting part starts. You are paying for the controller, the agent fleet, the storage, and the engineering time to patch all three. That last item is the largest line and the one nobody puts in the spreadsheet.

Two observations that matter more than the platform choice itself. First, the levers with the biggest effect on a CI bill are the same everywhere: cancel superseded runs, cache dependencies properly, skip workflows on paths that did not change, and stop running the full matrix on every draft commit. A team that does none of these will pay more on the cheapest platform than a disciplined team pays on the most expensive one.

Second, self-hosted runners are cheaper per minute and are not free. A pool of runners on a VPS provider like Hetzner, DigitalOcean or InterServer is inexpensive to rent and still needs image maintenance, autoscaling, disk cleanup and a security model. The break-even against hosted minutes is real, but it arrives later than people expect once you price the engineering hours honestly.

Security is where the differences stop being cosmetic

Every CI system is a machine that holds production credentials and executes code on demand. The platforms differ mostly in which mistakes they make easy.

• Untrusted pull requests. This is the classic GitHub Actions trap. The pull_request trigger runs fork code without access to your secrets. The pull_request_target trigger runs in the privileged context of the base repository, and if a workflow using it checks out the pull request’s code, an outside contributor can execute arbitrary code with your secrets in scope. GitHub’s own guidance is to avoid that trigger unless the workflow genuinely needs the privileged context, and never to check out untrusted code in it.
• Expression injection. Any attacker-controlled string, such as a branch name, issue title or commit message, that gets interpolated directly into a shell command is a code execution path. Bind it to an environment variable and reference the variable instead. This applies equally to GitLab, where the same values arrive as predefined CI variables.
• Self-hosted runners on public repositories. Don’t. A persistent runner that executes fork code is compromised for every subsequent job on it, including jobs that do have secrets. If you must self-host, use ephemeral runners that are destroyed after a single job, and keep public repositories on hosted runners.
• Mutable action and image references. Pin third-party actions to a full commit SHA. Tag repointing has been used in real supply chain compromises to reach thousands of repositories at once. The same logic applies to base images referenced by tag in GitLab jobs.
• Credential scope. Prefer short-lived federated credentials over long-lived secrets wherever the target supports it. GitHub and GitLab both issue OIDC tokens that cloud providers and HashiCorp Vault can exchange for short-lived access, and a token that expires in an hour is a much smaller incident than a static key that nobody rotates. Where you do use variables, mark them protected so they are only exposed on protected branches.
• Jenkins specifically. The controller holds the credentials store, so builds should never execute on it. Restrict agent-to-controller access, keep the plugin list as short as you can defend, and subscribe to the project’s security advisories, because plugin vulnerabilities are the most common way Jenkins installations get owned.

A decision procedure you can run in an afternoon

1. Follow platform gravity first. Where does the code live, and where will it live in three years? CI that lives next to the code is cheaper to operate than CI that has to be integrated with it. If that single question has a clear answer, it decides most cases and the rest of this list is confirmation.
2. Check whether a runner can reach the deploy target. Not the build. The deploy. If production sits behind a private network with no inbound path, you need self-hosted runners, a tunnel, or a system that already lives inside the perimeter. This is the requirement that most often overrides answer one.
3. Name the on-call owner. Write down who gets paged when the CI system itself is down at 2am on a Sunday. If the honest answer is nobody, choose a hosted platform. Jenkins and self-managed GitLab both need that name to exist.
4. Model the compute, not the sticker price. Take last month’s build minutes, split them by operating system and machine size, and apply each platform’s mechanics. macOS-heavy and Windows-heavy workloads change the answer significantly.
5. Inventory what the plugins actually do. If you are leaving Jenkins, go through the plugin list and mark each one as replaced by a native feature, replaced by a marketplace action, needs writing, or can be dropped. The “needs writing” pile is your real migration estimate.
6. Trial with your ugliest pipeline. Not the greenfield service. The one with the flaky integration test, the hardware dependency and the manual approval. Every platform handles a clean Node build beautifully. Only one of them will handle yours.

The hybrid pattern nobody puts on the slide

A very common real-world arrangement: GitHub Actions or GitLab CI runs build, test, scan and image publish, then a retained Jenkins instance handles deployment into restricted environments. It works, and for regulated organisations it is sometimes the only thing that does.

Be honest about what it costs, though. You now have two systems to patch, two places secrets live, and a handoff between them that needs its own monitoring. It should be a deliberate architecture with a documented boundary, not the accidental result of a migration that stalled. The tell is whether anyone can draw the handoff on a whiteboard without hedging.

Arguments that don’t survive contact

• “Jenkins is legacy.” It is old, which is not the same thing. It is actively maintained and still the only option for several categories of requirement. The real objection is the maintenance cost, so make that argument instead; it is stronger and it is true.
• “We’ll save money with self-hosted runners.” On per-minute compute, yes. Once you price the image maintenance, autoscaling and patching, the saving is much smaller than the spreadsheet suggested, and it only exists at volume.
• “GitLab CI is just GitHub Actions with different YAML.” The pipeline languages are comparable. The products are not. One is an event platform around a repository, the other is an integrated lifecycle platform. The difference shows up in permissions, scanning and audit, not in syntax.
• “We’ll stay vendor-neutral by keeping all logic in shell scripts.” Good instinct, incomplete. The scripts port. The triggers, permissions, secret injection, artifact handling and approval gates do not, and those are most of the migration work.
• “The bigger marketplace wins.” Marketplace size counts unreviewed third-party code you are about to execute with your credentials. It is a convenience and a supply chain liability at the same time. Treat it as both.

How I’d decide

• Code on GitHub, cloud deploy targets, no unusual compliance requirement: GitHub Actions, hosted runners, OIDC to the cloud provider, actions pinned to SHAs. This is the one I reach for first because it is the least infrastructure to own.
• Wanting one vendor for source, CI, registry and scanning, especially with compliance pressure: GitLab, and take self-managed seriously if data residency matters.
• Restricted networks, unusual hardware, or existing Jenkins that works and has an owner: keep Jenkins, but put the pipeline in a Jenkinsfile, the controller config in Configuration as Code, and the plugin list under version control. Jenkins pain is almost always click-configured Jenkins pain.
• Whichever you pick, wire pipeline duration and failure rate into whatever you already use for metrics, whether that is Grafana, Datadog or something homegrown. A CI system you cannot see is one you cannot tune, and slow pipelines are a cost problem and a morale problem at the same time.
• Do not migrate for aesthetics. Migrate because a specific requirement is unmet, and write that requirement down before you start.

Frequently asked questions

Is Jenkins still worth using?

Yes, for specific situations. Air-gapped networks, source control that is not GitHub or GitLab, hardware-attached builds, and orchestration of things that are not software builds are all cases where Jenkins is the practical answer. For a standard cloud-deployed web service with code on GitHub, a hosted platform will cost you less operationally. The deciding factor is whether you have someone who owns the controller.

Which is cheaper, GitLab CI or GitHub Actions?

It depends on your operating system mix and team size, because the two use different mechanics. GitHub applies multipliers so Windows and especially macOS jobs drain the included pool much faster than Linux. GitLab’s included compute pool is per namespace rather than per seat, so it does not grow as you hire. Both make self-hosted runners free of per-minute charges. Model your own last month of build minutes rather than trusting a general answer.

Can I use GitLab CI with a GitHub repository?

There are mirroring and integration options, but the pipeline features are built around GitLab-hosted repositories and the experience degrades away from that. If your code is staying on GitHub, treat GitLab CI as a poor fit rather than a configuration challenge.

How long does migrating from Jenkins to GitHub Actions or GitLab CI take?

Translating pipeline stages is usually days per service. The schedule is set by the parts that are not the pipeline: replacing plugin functionality, rebuilding the secret and identity model, provisioning runners with network access to deploy targets, and reproducing approval gates. Inventory those first, because that inventory is the estimate.

Are self-hosted runners safe?

They are safe for private repositories with trusted contributors, provided the runner is ephemeral and destroyed after each job. They are not safe for public repositories, because a fork pull request can execute code on a persistent runner and reach everything that runs on it afterwards. Keep public repositories on hosted runners.

Do I need to pin GitHub Actions to a commit SHA?

For any third-party action that runs with access to secrets, yes. Tags are mutable, and repointing a widely used tag to a malicious commit is a demonstrated attack path that has affected large numbers of repositories at once. Pinning costs you a dependency update per release and removes an entire class of compromise.

Is it reasonable to run more than one CI system?

Reasonable if it is deliberate, expensive if it is accidental. A documented split where hosted CI builds and tests while an internal system deploys into restricted networks is a legitimate architecture. A split that exists because a migration stalled halfway is a permanent tax with no owner.

The one thing worth remembering

Comparing GitLab CI vs GitHub Actions vs Jenkins on syntax and feature checklists produces a decision that feels rigorous and predicts almost nothing. The pipeline file is the part that ports. The runner fleet, the secret and identity model, the artifact storage and the network path to your deploy target are the parts that get rebuilt, and they are where the schedule and the risk actually live.

Pick the platform closest to where your code lives, unless a hard requirement about network access, data residency or unusual execution environments overrides it. Then spend your effort on the layer beneath the YAML, because that is the layer you will be living with.

Need help choosing or moving a CI pipeline?

This is a large part of what I do as a freelance DevOps engineer. Typical engagements around CI/CD platform decisions look like:

• A platform evaluation with your actual build minutes, operating system mix and deploy targets modelled, so the recommendation is arithmetic rather than opinion.
• Jenkins to GitHub Actions or GitLab CI migrations, including the plugin inventory, the secret and identity rebuild, and a staged cutover that does not leave you running two systems forever.
• Pipeline security review: fork pull request triggers, expression injection paths, action pinning, runner isolation and token permission scoping.
• Replacing long-lived cloud keys with OIDC federation on GitHub Actions or GitLab CI, including the trust policy and claim conditions on the cloud side.
• Self-hosted runner fleets that autoscale, stay ephemeral, and do not quietly fill their disks.
• CI cost reduction: caching, concurrency cancellation, path filters and matrix pruning, measured before and after.
• Jenkins hardening and Configuration as Code, so the controller is reproducible instead of a server nobody wants to touch.

If you want a second opinion, send me the pipeline file, a plugin list or a month of build minutes and I will tell you what I would actually do with it.

The post GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later appeared first on John Nessime.

None of that is a bug. AWS optimises the empty account for getting something running in ten minutes. The problem is that most accounts never get corrected afterwards, and eighteen months later there is production traffic sitting on top of decisions nobody made deliberately.

An AWS account baseline is the set of things you lay down before the first workload, so the account starts from a defensible posture instead of a blank console. This walks through the decisions that actually matter, why each one is the way it is, and what it costs. I have put a working reference implementation on GitHub — terraform-aws-baseline — so you can read the real Terraform rather than take my word for any of it.

What a baseline is, and what it is not

A baseline is not a landing zone in the Control Tower sense, and it is not a compliance framework. It does not make you SOC 2 compliant and nothing that fits in one repository will.

What it is: the boring foundational layer that everything else sits on. State storage, network segmentation, identity, and the guardrails that stop a careless afternoon becoming an incident. If you get these right once, every workload that lands afterwards inherits them for free.

The structure that works is three layers. A bootstrap stack that creates the state backend and runs exactly once. Reusable modules for the things every environment needs. Thin per-environment roots that compose those modules with different parameters — dev gets one NAT gateway and seven-day log retention, prod gets one per availability zone and a year.

Decision one: no long-lived AWS credentials, anywhere

Static access keys in CI are the single most common way AWS accounts get compromised. They live in a secrets store, they get copied into a second one, somebody echoes one into a build log, and nobody rotates them because rotation breaks things.

GitHub Actions can assume an AWS role using a short-lived OIDC token that GitHub signs and AWS verifies. There is no key to store, nothing to leak, and nothing to rotate.

The important part is the trust policy, and one word in it decides whether this is a security improvement or a vulnerability:

data "aws_iam_policy_document" "ci_assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    # StringEquals, not StringLike. This is the whole ballgame.
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values = [
        "repo:${var.github_org}/${var.github_repo}:ref:refs/heads/${var.github_branch}"
      ]
    }
  }
}

Swap StringEquals for StringLike and add an asterisk, and you have widened that trust from one branch of one repository to whatever the wildcard covers. Get sloppy enough — a bare repo:* — and any repository on GitHub can assume your role. This is not theoretical; it is a recurring finding in real audits.

Scope it to one repository and one branch. If you need more, add more statements, explicitly.

Decision two: treat Terraform state as a secret, because it is

Terraform state contains every attribute of every resource it manages. Database passwords, generated keys, connection strings. Marking a variable sensitive redacts it from console output and changes nothing about what is written to the state file.

So the state bucket gets four things: server-side encryption with a customer-managed KMS key that rotates, versioning so a corrupted state can be recovered, all four public access block flags, and a bucket policy that refuses plaintext connections outright.

resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

data "aws_iam_policy_document" "deny_insecure_transport" {
  statement {
    sid       = "DenyInsecureTransport"
    effect    = "Deny"
    actions   = ["s3:*"]
    resources = [
      aws_s3_bucket.state.arn,
      "${aws_s3_bucket.state.arn}/*",
    ]

    principals {
      type        = "*"
      identifiers = ["*"]
    }

    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }
}

All four public access block flags, not two. They cover different paths — ACLs and bucket policies, existing objects and future ones — and setting a subset leaves a gap that reads as protected on a dashboard.

There is a chicken-and-egg problem here that catches everyone: the stack that creates the state bucket cannot store its own state in a bucket that does not exist yet. You run it once with local state, then migrate:

cd bootstrap
terraform init
terraform apply
terraform output          # bucket, lock table, KMS key, region

# Now move bootstrap's own state into the bucket it just created
terraform init -migrate-state

Decision three: shut the default security group

Every VPC gets a default security group that permits all traffic between members and all outbound traffic. Anything launched without an explicit security group lands on it. That is a quiet failure mode — nothing errors, the instance works, and it has more network reach than anyone intended.

Terraform can adopt and empty it:

resource "aws_default_security_group" "default" {
  vpc_id = aws_vpc.this.id

  # No ingress and no egress blocks. Terraform strips AWS's rules,
  # so anything that accidentally lands here can reach nothing.
}

Four lines, and misconfiguration becomes loud instead of silent.

Decision four: three subnet tiers, and the bottom one has no way out

The standard two-tier split — public and private — leaves databases in the same tier as application servers. Both can reach the internet through NAT. That is fine right up until something in the data tier is compromised and quietly exfiltrates over 443, which looks like ordinary egress traffic.

• Public — NAT gateways and load balancers. Route to the internet gateway.
• Private — workloads. Outbound through NAT, no inbound from the internet.
• Database — data stores. No default route at all.

The third tier is enforced by absence rather than by a rule:

resource "aws_route_table" "database" {
  vpc_id = aws_vpc.this.id

  # Deliberately no 0.0.0.0/0 route. Not to the IGW, not to NAT.
  # Nothing here can reach the internet, and nothing can reach it.
}

No security group to misconfigure, no NACL to get wrong. The route simply does not exist. That is a much stronger guarantee than a rule someone can widen during an incident and forget to narrow afterwards.

Decision five: gateway endpoints, the one genuinely free win

S3 and DynamoDB gateway endpoints cost nothing. Not “cheap” — zero. They add a route so traffic to those services stays on the AWS backbone rather than going out through NAT.

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.this.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"

  route_table_ids = concat(
    aws_route_table.private[*].id,
    aws_route_table.database[*].id,
  )
}

Two benefits, and the second is the one people miss. Traffic to S3 never traverses NAT, so you stop paying per-gigabyte processing on it. And the isolated database tier can still reach S3 and DynamoDB despite having no internet route, because the endpoint is a route within the VPC rather than a path out of it.

Your Terraform state lives in S3. Without this endpoint, every plan and apply pushes state through the NAT gateway and you are billed for the privilege.

Decision six: SSM Session Manager, not SSH

Port 22 open anywhere means key pairs to distribute, a bastion host to patch, and an audit trail that is whatever auth.log happens to contain.

resource "aws_iam_role_policy_attachment" "ssm_core" {
  role       = aws_iam_role.instance.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

That one attachment gets you shell access with no inbound ports, no key pairs, no bastion. Access is IAM-authorised, so revoking someone is a permissions change rather than a key rotation, and every session can be logged centrally.

Worth knowing the caveat: SSM agents reach the SSM endpoints over the internet by default, so instances in the private tier still need NAT for this. Removing NAT entirely means adding interface endpoints for ssm, ssmmessages and ec2messages, and those cost roughly seven dollars a month per availability zone each.

Decision seven: pin actions to commit SHAs

A Git tag is a movable pointer. If an action’s repository is compromised, an attacker can repoint v4 at whatever they like, and every workflow referencing that tag runs it on the next build — with your OIDC role attached.

steps:
  # A tag can be moved. A commit SHA cannot.
  # Dependabot updates these pins and shows you the diff.
  - uses: actions/checkout@<full-40-character-sha>   # v4.2.2

The comment matters as much as the pin. Without it nobody can tell at a glance which version they are looking at, and the pin becomes something people update blindly.

What it actually costs

Almost all of it is NAT gateways. Everything else is rounding.

• NAT gateways — roughly $32 a month each, plus about $0.045 per gigabyte processed. One in dev, one per availability zone in prod.
• VPC, subnets, route tables, internet gateway — free.
• S3 and DynamoDB gateway endpoints — free.
• KMS keys — about a dollar a month each.
• Flow logs — around $0.50 per gigabyte ingested. Retention is the lever: seven days in dev, a year in prod.
• State bucket and lock table — cents.

Ballpark, that lands around $35 a month for dev and $105 for prod, dominated entirely by NAT. If you only need the isolated tiers and nothing requires outbound internet, dropping NAT takes dev close to free.

Per-AZ NAT in production is an availability decision, not a security one. A single NAT gateway is a single point of failure for outbound traffic across every AZ that routes through it.

Gates that run before any credentials exist

Most of what goes wrong in Terraform is catchable statically. Formatting, syntax, provider misuse, and known-insecure patterns can all be found without touching an AWS account, which means they can run on every commit and in CI without the pipeline holding any cloud credentials at all.

# On every commit, locally
pre-commit install        # gitleaks, fmt, validate, tflint, checkov, terraform-docs

# The same gates CI runs
make all                  # fmt + validate + tflint + checkov

Running the identical command locally and in CI is the part that matters. When those two drift, people stop trusting the local check and start pushing to see what the pipeline says, and the fast feedback loop you built quietly stops existing.

Note also what is missing: there is no terraform plan in that pipeline. Plan needs real credentials, and a static-only pipeline is one that cannot leak anything. A production setup would add a plan job running under the OIDC role and posting to the pull request — but that is a deliberate step up in blast radius, not a default.

What a baseline like this deliberately leaves out

Being clear about scope is more useful than pretending completeness. The reference implementation covers preventative controls and stops there. Missing, on purpose:

• Detective controls. No CloudTrail, Config, GuardDuty or Security Hub. A complete account baseline includes them; this is the preventative half.
• Compute. There is an instance role and profile, but nothing uses them. No ASG, no ECS or EKS, no load balancer.
• Interface endpoints. Only the free gateway ones. Fully private SSM needs paid interface endpoints.
• Multi-region and DR. Versioning covers state recovery. There is no cross-region replication and no failover story.
• Policy-as-code tests. Checkov catches known misconfigurations; there are no Terratest or Conftest assertions on module contracts.

A baseline you can describe the edges of is more useful than one advertised as complete. You know exactly what you still have to build.

Common mistakes

• Using StringLike with a wildcard in an OIDC trust policy. Scope to one repository and one branch.
• Assuming sensitive = true keeps values out of state. It does not.
• Setting two of the four S3 public access block flags and considering the bucket locked down.
• Leaving the default security group as AWS shipped it.
• Putting databases in the same tier as application servers because two tiers felt like enough.
• Skipping gateway endpoints and paying NAT processing charges on your own state file.
• Referencing actions by tag and assuming a tag is immutable.
• Committing backend.hcl or terraform.tfvars. State bucket names embed the account ID.

Frequently asked questions

Is an AWS account baseline the same as a landing zone?

No. A landing zone in the Control Tower sense governs many accounts — organisational units, service control policies, account vending. A baseline is what you lay down inside a single account. Baselines are what you need when you have one or two accounts and Control Tower would be enormous overkill.

Why not just use Control Tower?

Control Tower is a good answer for a multi-account organisation with governance requirements. It is a lot of machinery for a team running a handful of accounts, and it hides the decisions rather than making them legible. Writing the baseline yourself means you can read every choice and change any of them.

Do I really need a separate database subnet tier?

It costs nothing and closes an entire exfiltration path. If your data tier has no route to the internet, a compromised database cannot phone home regardless of what the attacker does with it. Two tiers is not wrong, but the third is close to free.

How do I cut the NAT gateway cost?

Use one NAT for non-production instead of one per AZ. Add gateway endpoints so S3 and DynamoDB traffic bypasses NAT entirely. Beyond that, work out what genuinely needs outbound internet — often less than you assume — and consider dropping NAT for environments that do not.

Can I use this with OpenTofu?

Yes. It is standard HCL against the AWS provider, so it runs on either. Swap terraform for tofu in the commands and everything else is the same.

Where does the OIDC provider itself live?

It is account-global, so only one environment can create it and the others reference it. That is workable but awkward. In a real organisation it belongs in a separate account-level stack that runs before any environment.

Wrapping up

None of the decisions in an AWS account baseline are individually clever. Scope the trust policy. Encrypt the state. Empty the default security group. Give the data tier no way out. Take the free endpoints. Use SSM instead of SSH. Pin your actions. Each one is a few lines of Terraform.

What makes them valuable is doing them before the first workload arrives, because retrofitting network segmentation into a running account is a project, and doing it at the start is an afternoon.

The full implementation is on GitHub at JohnNessime/terraform-aws-baseline, MIT licensed. It is a reference implementation rather than something that has served production traffic — a worked example of how I would lay an account down, with the reasoning written out in the docs. Read it, disagree with parts of it, take what is useful.

Need a baseline laid down properly?

I work with teams on AWS foundations — account structure, network segmentation, IAM that does not rely on long-lived keys, Terraform layouts that survive more than one engineer, and CI pipelines that catch problems before they reach an account.

If you would rather have this done once and done right, you can hire me on Upwork.

The post A Secure-by-Default AWS Account Baseline in Terraform appeared first on John Nessime.