Then someone in finance asks what the reporting pipeline costs. You open Cost Explorer, group by Project, and over half the bill lands in a bucket called “No tag key”.

Nothing is broken. The tags are there. They just aren’t on the bill, because tagging a resource and activating a cost allocation tag are two separate operations, and AWS never warns you when you’ve done only the first.

This post covers getting AWS cost allocation tags to cover a messy account: the activation gap, what backfill will and won’t repair, where coverage leaks even with decent discipline, and a retrofit order that converges.

A resource tag and a cost allocation tag are not the same thing

A resource tag is metadata on the resource. A cost allocation tag is a billing dimension. Getting the second means taking a key that already exists on your resources and explicitly switching it on in the Billing and Cost Management console.

Two details make this bite. First, activation lives in the management account. From a member account, the page tells you tags are managed at the payer level and that’s the end of it. Plenty of teams have a good tagging standard and no access to the one screen that makes it count. Related trap: an account moving between organizations has its activated keys reset to inactive.

Second, the lag. AWS documents up to 24 hours for a newly applied key to appear on the activation page, and up to another 24 hours after you activate before it takes effect. The loop between “we agreed a standard” and “I can see it in Cost Explorer” is two days at best.

Past a handful of keys, drive it from the API. Listing the inactive ones is the fastest way to find what you forgot:

# Every key AWS knows about that is NOT currently billing-active
aws ce list-cost-allocation-tags --status Inactive

# Switch a set of them on
aws ce update-cost-allocation-tags-status --cli-input-json '{
  "CostAllocationTagsStatus": [
    {"TagKey": "Project",     "Status": "Active"},
    {"TagKey": "Environment", "Status": "Active"},
    {"TagKey": "Owner",       "Status": "Active"}
  ]
}'

User-defined keys appear in the report with a user: prefix and AWS-generated ones carry the reserved aws: prefix, so they never collide. There’s a cap on active keys, raisable through Service Quotas, which you almost certainly don’t need to raise.

What the backfill repairs, and what it quietly doesn’t

Activation used to be strictly forward-looking, so activating late permanently orphaned everything before it. That’s no longer true. The management account can ask AWS to reapply current activation status across up to twelve months of historical cost data.

# Must be the first day of a month, no earlier than twelve months back
aws ce start-cost-allocation-tag-backfill 
  --backfill-from 2025-01-01T00:00:00Z

• Start date must be the first day of a month, and can’t reach back more than twelve months.
• One request per 24 hours, and you can’t submit another while one is processing.
• It backfills activation status, not tag values.

That last point catches people. If the resource wasn’t carrying the tag when the cost was incurred, backfill gives you nothing for those months. It fixes “I forgot to activate.” It does not fix “I forgot to tag.” A tag applied in June and activated in November backfills to June and no further, whatever start date you hand it.

It also cuts both ways. Backfill applies the current status of every key, so a key sitting inactive right now gets stripped out of historical data too. If you’re pruning an overgrown tag list, backfill before you deactivate the old keys, not after, or you’ll erase the history you were trying to keep.

Where AWS cost allocation tags actually leak coverage

With activation sorted and every Terraform-managed resource tagged, a stubborn slice of the bill still refuses to attribute. It matters which slice, because the fixes have nothing in common.

Charges no resource can carry

Support charges, refunds, credits, tax, some Marketplace subscriptions. There’s no resource to hang a tag on, so tagging discipline never touches them. On accounts with a large support plan or heavy credit activity, that’s a visible fraction of the invoice.

The answer is account tags. AWS added support for using AWS Organizations account tags as a cost allocation dimension, applying to all metered usage inside the tagged account and flowing into Cost Explorer, Budgets, Cost Anomaly Detection and Cost Categories. Because attribution happens at the account level, it picks up exactly what resource tags can’t reach.

This is recent enough that most tagging guides predate it, and it’s the highest-value item here if your accounts already map roughly onto teams.

Resources created by other resources

This is the quiet one, and usually the biggest surprise gap in an otherwise well-run account. Terraform’s provider-level default_tags applies to resources Terraform creates. It does not reach the EC2 instances an Auto Scaling group launches, because Terraform didn’t create those, the ASG did. Your ASG is tagged. Your instances aren’t. Same for volumes attached at launch, ENIs, snapshots and AMIs.

data "aws_default_tags" "current" {}

resource "aws_autoscaling_group" "app" {
  # ... other config ...

  # Push provider default tags down to launched instances
  dynamic "tag" {
    for_each = data.aws_default_tags.current.tags
    content {
      key                 = tag.key
      value               = tag.value
      propagate_at_launch = true
    }
  }
}

resource "aws_launch_template" "app" {
  # ... other config ...

  tag_specifications {
    resource_type = "instance"
    tags          = data.aws_default_tags.current.tags
  }

  tag_specifications {
    resource_type = "volume"
    tags          = data.aws_default_tags.current.tags
  }
}

For a standalone instance, volume_tags on aws_instance covers its attached volumes. General rule: whenever one AWS resource provisions another on your behalf, assume tags don’t propagate until you’ve checked the created resource in the console, not the plan output.

Shared infrastructure that belongs to everyone

NAT gateway data processing, VPC interface endpoints, a Transit Gateway, a shared load balancer, the central logging bucket. You can tag all of these perfectly and the tag says Project=platform, which is accurate and useless for chargeback.

Transit Gateway is worth calling out because it supports tagging the gateway resource itself, so data processing and transfer charges pick up the tag rather than only attachment hours. That gap used to swallow a lot of network spend.

Beyond that, stop chasing tags and do allocation math instead. Cost Categories support split charge rules that redistribute a shared bucket across your other categories, using proportional splits, fixed percentages or an even division. Proportional is the sane default; fixed percentages are for when finance already agreed a number and the report has to match it.

Container spend hiding behind one instance tag

A shared EKS or ECS cluster is one tagged compute footprint running fifteen teams’ workloads. The instance tag gives you the total and nothing about who caused it. Split cost allocation data is the native answer: opt in and the cost and usage export gains rows breaking cluster cost down to individual pods or tasks. It won’t account for idle headroom you’re deliberately carrying, but it beats one line item labelled “the cluster”.

Case drift and near-duplicate keys

Keys and values are case-sensitive. Environment, environment and ENV are three distinct keys producing three columns, each holding a partial and misleading picture. Tag Policies in AWS Organizations exist for this: define the key including its capitalization, define permitted values, and get an org-wide compliance report. Cheapest governance on this list, because it costs nobody a deploy.

A retrofit order that converges

Retrofits stall because people start at the bottom, sweeping untagged resources by hand while the creation path keeps producing more. This order avoids that.

1. Measure first. Group Cost Explorer by your intended key and record what percentage of spend falls into “No tag key”. That’s your baseline and your only honest progress metric.
2. Cut the key set down. Three to five keys: owner, environment, and a project or cost centre. Coverage on five beats partial coverage on thirty.
3. Activate immediately, even with terrible coverage. Both clocks start now: the activation delay and the rolling twelve-month backfill window. There’s no downside to activating a key that’s barely applied.
4. Rank the gap by spend, not resource count. Sort untagged line items by cost descending. It’s usually a few NAT gateways, a couple of oversized databases and one forgotten bucket. Ten thousand untagged security groups cost nothing.
5. Fix the creation path before the backlog. IaC defaults, launch template tag specifications, ASG propagation. Sweeping first means sweeping again next month.
6. Sweep the stragglers in bulk. The Resource Groups Tagging API works across services and regions, which beats going service by service.
7. Backfill, then re-measure. Compare against step one. If the number barely moved, you fixed the wrong resources.

# Find what already carries a Project tag, to diff against inventory
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project

# Apply tags to a specific set of ARNs
aws resourcegroupstaggingapi tag-resources 
  --resource-arn-list arn:aws:rds:eu-west-1:111122223333:db:reporting-01 
  --tags Project=reporting,Environment=prod

Run it against a couple of ARNs and confirm in the console before looping over a list. Bulk tagging is trivially easy to point at the wrong account.

Enforcing tags without becoming the tag police

Four mechanisms, in increasing order of blast radius. Most teams need the first three and should be careful with the fourth.

• IaC defaults. Provider-level default tags plus required module variables. Cheapest, catches the majority, nobody thinks about it.
• CI validation. A linter rule that fails the pull request on missing required tags. The failure lands on the person who can fix it, when they can fix it.
• Detective controls. The AWS Config managed rule for required tags flags non-compliant resources without blocking anything. Tells the truth, breaks nothing.
• Preventive controls. A service control policy denying creation when a required key is absent, keyed on a null check against the request tag.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyRDSCreateWithoutProjectTag",
      "Effect": "Deny",
      "Action": "rds:CreateDBInstance",
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/Project": "true"
        }
      }
    }
  ]
}

The honest trade-off: service control policies are where tagging programmes go to die. Not every API supports tag-on-create, and not every legitimate workflow passes tags through the way you expect. Roll a broad deny across the org on day one and you break somebody’s deployment on day two, the ticket lands on you, and the policy is gone by day three along with everyone’s appetite for the project.

Scope it to one service at a time, starting in sandbox. I’d rather run Config plus Tag Policies for a quarter and fix the top three offenders in the IaC. The deny is a finishing move, not an opening one.

Measuring coverage so it doesn’t rot

Cost Explorer gives the headline number. For a per-service breakdown, query the cost and usage export in Athena. One caveat: tag column naming differs between export versions, older ones giving a flattened column per activated key and newer formats a map column. Run a DESCRIBE first rather than guessing.

-- Confirm the actual column names first
DESCRIBE your_cur_database.your_cur_table;

-- Then: untagged spend, ranked by service
SELECT   line_item_product_code,
         SUM(line_item_unblended_cost) AS untagged_cost
FROM     your_cur_database.your_cur_table
WHERE    your_project_tag_column = ''
  AND    line_item_unblended_cost > 0
GROUP BY 1
ORDER BY 2 DESC
LIMIT    20;

That ranked list is your work queue; everything below the top twenty is noise. The list call also returns a last-used timestamp per key, the cleanest way to find keys nobody has applied to anything in months.

If you’d rather not maintain the dashboard yourself, this is a fair place to buy rather than build. Grafana Cloud sits on Athena comfortably if you already run Grafana, and platforms like CloudZero, Vantage and nOps handle allocation modelling out of the box. Their real value is the shared-cost and commitment maths. Their real cost is another vendor between you and your billing data, and several price against a percentage of the spend they watch, so run that number first.

Troubleshooting

• The key isn’t in the activation list. It must exist on at least one resource first, and propagation takes up to a day. Past that, check you’re in the management account and the key matches exactly, including case.
• Activated, but everything still shows untagged. Give it a full billing cycle. A key switched on mid-month leaves the first half of that month untagged until you backfill.
• Tags stopped working after an account move. Activation doesn’t survive a move between organizations. Reactivate from the new management account, then backfill.
• Cluster costs still land on one team. Instance tags can’t split a shared cluster. Opt into split cost allocation data, or treat the cluster as a shared cost in a split charge rule.
• You’ve hit the active key limit. Almost always too many keys rather than a real need for a quota increase. Use last-used timestamps to find dead ones, and backfill before deactivating.

Common mistakes

• Assuming a tagged resource is a tagged cost. It isn’t until you activate the key.
• Waiting for good coverage before activating. Activate first, improve coverage second, backfill third.
• Deactivating stale keys and then backfilling, which wipes their history out of the data.
• Measuring coverage by resource count instead of by spend.
• Rolling out a broad tag-enforcing SCP in production before anyone has read a compliance report.
• Trusting default_tags to reach resources Terraform didn’t directly create.
• Putting anything sensitive in a tag value. Tags surface in billing exports that finance and third-party tools read.

Best practices

• Keep the mandatory set to three to five keys, with defined allowed values rather than free text.
• Pin casing in the standard and enforce it with Tag Policies, because AWS won’t do it for you.
• Tag accounts as well as resources. Account tags cover charges resource tags structurally cannot.
• Treat coverage as a tracked percentage of spend with an owner and a monthly trend, not a project that finishes.
• Accept an irreducible remainder. Chasing the last few percent by hand costs more than it returns.
• Put allocation rules in Cost Categories rather than inventing tag values that encode a split.

Frequently asked questions

Are AWS cost allocation tags retroactive?

Partly. Activation applies going forward, but the management account can request a backfill reapplying current activation status across up to twelve months. The catch is that the resource must have carried the tag at the time, so backfill recovers late activation, never late tagging.

Why don’t my tags appear in Cost Explorer?

In order of likelihood: the key was never activated in the Billing console; you’re looking from a member account; the activation delay hasn’t elapsed; or the key differs in case from the one you activated.

How many cost allocation tags can I activate?

There’s a documented per-account cap on active keys, raisable through Service Quotas, plus a separate per-resource cap on total tags. Check current numbers in the AWS billing quotas docs. In practice the binding limit is human: past roughly a dozen keys, consistency collapses.

Should I use account tags or resource tags?

Both, for different jobs. Account tags handle top-level attribution and cover untaggable charges like support, credits and refunds. Resource tags handle granularity inside an account. If your accounts already map cleanly to teams, account tags are the cheaper win and worth doing first.

Is this worth doing on a small account?

Scale the effort to the shape of the bill. If you run a handful of services under one account and can name every resource from memory, account tags plus two resource keys is plenty. Worth noticing too that a fixed-price VPS from a provider like Contabo or InterServer has trivially perfect cost allocation, because the bill is one predictable line. Elastic infrastructure buys flexibility and pays for it in attribution work.

How do I allocate shared costs like NAT gateways?

Not with tags. Put them in a Cost Categories split charge rule and redistribute proportionally across consuming categories. Tag the shared resource to identify the bucket, then let the rule do the allocation.

The one thing worth remembering

AWS cost allocation tags fail silently, and that’s what makes them frustrating. Nothing errors. No alarm fires. Your resources look correctly tagged in every console you check, the bill quietly disagrees, and you find out when someone asks a question you can’t answer.

So check one thing now: open the cost allocation tags page in your management account and see whether the keys you’ve been diligently applying are actually switched on. If they aren’t, activate them today, then backfill. That twelve-month window rolls, and every month you wait is history you can’t recover.

Everything else here is incremental. Coverage is a percentage you improve, not a state you reach.

Need help getting your AWS bill to explain itself?

Cost attribution is unglamorous and easy to postpone until the quarter someone actually needs the numbers. If your account is already messy, it goes faster with someone who has untangled one before. I help with:

• Auditing tag coverage as a percentage of spend, broken down by service, so you know what you’re actually fixing
• Designing a minimal tag schema and the Tag Policies to keep it consistent across an organization
• Closing propagation gaps in Terraform and CloudFormation, around Auto Scaling groups, launch templates and attached storage
• Setting up Cost Categories with split charge rules so shared infrastructure stops distorting every team’s number
• Building cost and usage export pipelines in Athena, partitioned so the dashboard isn’t expensive to load
• Planning a staged tag enforcement rollout that doesn’t break anyone’s deployment pipeline

Send me a Cost Explorer screenshot grouped by your main tag key, or the output of a list call against your active tags, and I’ll tell you where the gap actually is.

The post AWS Cost Allocation Tags: Getting Real Coverage on a Messy Bill appeared first on John Nessime.

That question is what actually bites when you run infrastructure for twenty clients. Not a breach. An audit, or a resignation. Someone gives notice on a Friday and you spend Monday working out which accounts they still hold keys to, because the credential everybody shared was never attached to a person. You can’t revoke a person from a password. You can only rotate it and hope you got all twenty.

This post covers how to design client AWS account access so that humans authenticate once against something you control, every client account holds a role that trusts exactly one thing, and every API call in a client’s CloudTrail can be traced back to a named engineer. It’s organised by failure family, because the mechanics are well documented and the failure modes are not.

The shape: one identity plane, twenty separate trust decisions

The instinct that causes all the trouble is putting credentials in the middle. Client gives you a login, you store it somewhere shared, engineers take it out when they need it. Everything downstream of that decision is damage control.

The shape that works is boring. One AWS account whose only job is identity. Your engineers exist there, once, in IAM Identity Center or federated from whatever directory you already pay for. Each client account holds an IAM role whose trust policy names one principal in that account and requires an external ID. Nobody stores a client credential, because there is no client credential. There’s a role your engineers can ask STS for a short-lived session against, and a client who can delete that role whenever they like.

There is a second design worth knowing about: inviting client accounts into your own AWS Organization. It’s genuinely better in some ways. You get service control policies, resource control policies, and centralised root access management across the estate, all of which are organisation-scoped features you simply cannot use on an account you don’t own. It’s also a much bigger commitment. You take on consolidated billing, the client’s spend flows through you, and pulling an account back out of an organisation at the end of a contract is a real project rather than a policy change.

For most consultancies and managed service providers, the client keeps their own organisation and you get a role. That’s the model the rest of this assumes.

Failure family 1: credentials you can’t revoke

The shared IAM user with an access key is the default because it works immediately. It keeps working right up until one of three moments: an engineer leaves, a client asks who did something, or a laptop gets stolen.

Access keys have no expiry. They’re bearer tokens with an indefinite lifetime, and the only way to invalidate one is to go into each account and delete it. At twenty accounts that’s twenty deliberate actions performed under time pressure, and the one you forget is the one that matters. A role session, by contrast, expires on its own. That’s the entire argument.

A shared password manager still earns its place, just not for admin keys. Bitwarden or 1Password is the right home for per-client external IDs, account numbers, role ARNs and the client-side contacts you need when something breaks at two in the morning. Metadata, not credentials.

Failure family 2: a trust policy that trusts more than you meant

This is the invisible one. The role gets created, the access works, everyone moves on, and the trust policy quietly grants far more than anyone intended.

Two mistakes travel together. The first is naming your account root in the principal. Writing arn:aws:iam::111122223333:root does not mean “the root user of my account.” It means any principal in that account that has permission to call AssumeRole. Every engineer, every Lambda function, every CI runner, every future role you haven’t created yet. It delegates the decision to your own account’s IAM, which is fine when your IAM is tidy and quietly catastrophic when it isn’t.

The second is skipping the external ID. AWS calls the problem it solves the confused deputy: you are a third party that holds the ability to assume roles in many customer accounts, and without a shared secret in the condition, one customer who learns another customer’s role ARN could persuade your systems to act against it. The external ID is the value that makes each trust relationship specific rather than generic.

Generate it yourself, one per client, as a UUID or similar unguessable value. The client must not be the one choosing it, because if they can choose it they can choose someone else’s, and the control does nothing.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/MspOperationsAccess"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02"
        }
      }
    }
  ]
}

The principal names one role, not the account. If that role is compromised you have one thing to fix. If the account is named, you have an inventory problem you didn’t know you had.

Failure family 3: CloudTrail says the role, not the human

You’ve done everything above and a client emails asking who deleted the staging load balancer. You open their CloudTrail and every event attributes to the same assumed role. The role is shared by your whole team by design, so the log tells you your company did it. That’s not what they asked.

There are two knobs and they are not equivalent. The role session name is chosen freely by whoever calls AssumeRole, appears in the assumed-role ARN, and is trivially set to anything. It’s useful, but it’s a label, not evidence. Source identity is the one that carries weight: once set on a session it cannot be changed, it persists across chained role sessions, and it lands in CloudTrail in the session context of every subsequent call.

The catch is that the role’s trust policy has to permit sts:SetSourceIdentity alongside sts:AssumeRole, and you can require it with a condition so a session without one is refused outright.

"Action": [
  "sts:AssumeRole",
  "sts:SetSourceIdentity"
],
"Condition": {
  "StringEquals": {
    "sts:ExternalId": "6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02"
  },
  "StringLike": {
    "sts:SourceIdentity": "*@yourcompany.com"
  }
}

If your engineers arrive through SAML federation, the identity provider can populate source identity from a directory attribute such as email, which means nobody has to remember to pass it. For direct CLI use it’s an explicit flag:

aws sts assume-role 
  --role-arn arn:aws:iam::444455556666:role/MspEngineerAccess 
  --role-session-name acme-investigation 
  --external-id 6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02 
  --source-identity jane.doe@yourcompany.com

Getting this right is what turns “we take security seriously” into a sentence you can back with a log query. It is also, in my experience of what clients actually ask for, the single most persuasive thing on the list.

Failure family 4: one identity, twenty blast radii

A hub account is a concentration of risk. That’s the trade you’re making, and it’s worth making, but only if the hub is held to a higher standard than the accounts it reaches into.

Practically: not every engineer gets permission to assume every client’s role. Group permissions in the hub so the people on the Acme engagement reach Acme and nothing else. Use per-client role names rather than one shared name across all twenty, so a policy in the hub can’t wildcard its way into a client you stopped working with a year ago.

Tier the roles by what they’re for, not by who’s asking. A read-only role for investigation covers most of the work most days. A change role carries the write permissions the engagement actually needs. Break-glass sits separate, with a short maximum session duration and an alarm on every use.

Two extras are worth knowing about even if you don’t reach for them immediately. Session policies let the caller narrow a session further at assume time, which is useful for automation that only needs one bucket. Permissions boundaries cap what any role or user created inside the client account can do, which matters when part of your job is creating IAM there.

If your team already routes through a business VPN or zero trust gateway with stable egress addresses, a source IP condition on the trust policy is a cheap extra layer. NordLayer, Tailscale and Cloudflare’s zero trust products all give you predictable egress. Be honest about the cost, though: the moment something runs from CI, a client’s network or a laptop on hotel wifi, an IP condition becomes a support ticket. I’d add it once everything else works, not before.

Failure family 5: standing administrator access nobody remembers granting

Somewhere in the twenty accounts there is a role with AdministratorAccess attached, created during a migration, described in the ticket as temporary. It’s been eighteen months.

Here’s the honest part: AWS does not give you a native just-in-time approval workflow for cross-account role assumption. There’s no built-in “request four hours of write access, get approved in Slack, access expires automatically” for this pattern. You either build that on top of STS and your ticketing system, or you accept standing access and compensate with tight session durations, aggressive monitoring and a genuine periodic review. Both are defensible. Pretending you have JIT when you have a wiki page is not.

The cheap version that actually helps: set the maximum session duration on client roles to something short, alert on assumption of the break-glass role, and run a quarterly pass over service last accessed data to strip permissions nobody has used. IAM can generate that report per role.

# Kick off the report, then fetch it with the returned job ID
aws iam generate-service-last-accessed-details 
  --arn arn:aws:iam::444455556666:role/MspEngineerAccess

aws iam get-service-last-accessed-details --job-id JOB_ID_FROM_ABOVE

Run that in each client account and the conversation about narrowing permissions stops being theoretical. You’re not guessing what the engagement needs; you’re reading it.

Failure family 6: offboarding that only half happens

The whole point of the hub is that offboarding is one action. Disable the person in your directory and every path into every client account closes at once, because there was never a credential in a client account with their name on it.

That’s true only if nothing crept in sideways. Things that creep in sideways:

• IAM users created directly in a client account during an incident, because it was faster
• Access keys minted under a role by automation and never inventoried
• A client’s own console user shared with your team “just for the migration”
• Third party tools connected with their own cross-account roles that outlived the trial
• A departing engineer’s laptop with cached SSO tokens and stale profiles still on disk

IAM Access Analyzer is the tool that finds most of the first four. Its external access findings show every principal outside the account that can reach into it, which is exactly the list a client wants to see, and it’s also the list that reveals the monitoring vendor nobody remembers signing up for. For the laptop, whatever your hardware disposal process is, make sure it involves an actual wipe rather than a factory reset; tools like O&O SafeErase exist for that reason and are cheap next to the alternative.

Rolling client AWS account access out across twenty accounts

Doing this by hand twenty times guarantees twenty subtly different configurations. Make it a template and a checklist.

1. Write one CloudFormation template that creates the role, its trust policy and its permissions, parameterised by external ID and your operations account ID.
2. Generate a unique external ID per client and store it with the client’s account metadata, not in the template.
3. Send the client a launch stack link rather than asking for credentials. They deploy it in their own account, in their own console, and can read exactly what they’re granting.
4. Record the account ID, role ARN, external ID location, engagement scope and review date in a single inventory file in version control.
5. Generate everyone’s local AWS profiles from that inventory so nobody hand-edits a config file.
6. Verify each one with a call that proves identity and nothing else before you touch anything real.

The generated profile chains from your federated session into the client role. The AWS CLI reads these from the shared config file and handles the assume-role call for you:

# ~/.aws/config
[profile msp-hub]
sso_session = company
sso_account_id = 111122223333
sso_role_name = MspOperationsAccess
region = eu-west-1

[profile acme-readonly]
role_arn = arn:aws:iam::444455556666:role/MspEngineerReadOnly
source_profile = msp-hub
external_id = 6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02
role_session_name = jane-doe
duration_seconds = 3600
region = eu-west-1

Then the two-second sanity check that has saved me from more mistakes than any other command:

aws sts get-caller-identity --profile acme-readonly

It returns the account ID and the assumed-role ARN. Read the account number before you run anything destructive. Twenty profiles in one config file is exactly the situation where muscle memory picks the wrong one.

For scheduled work that runs without a human, the same trust relationship serves a Lambda function or an EC2 instance role in your hub. A low-cost VPS from Contabo or InterServer will also do the job, but note the trade: an instance profile inside AWS gets rotated credentials automatically, while a box outside AWS needs a long-lived credential to bootstrap from, which reintroduces exactly what this design removes. Inside AWS is the cleaner answer.

Troubleshooting the failures you will actually hit

AccessDenied on AssumeRole

Three separate causes produce nearly the same message, and they need different fixes. Either the client’s trust policy doesn’t name your principal, or the external ID you sent doesn’t match what the condition expects, or your own identity in the hub lacks permission to call AssumeRole on that role ARN. Check them in that order, because the first two require the client and the third doesn’t.

Switch Role in the console silently fails

This one wastes an afternoon if you don’t know it. The console’s switch role feature does not support roles that require an external ID. It’s a documented limitation, not a bug in your policy. Engineers who need console access to those accounts should reach them through federation rather than the switch role box.

Sessions expiring after an hour when you asked for longer

When one role session assumes another role, that’s role chaining, and a chained session is capped at one hour regardless of the role’s maximum session duration setting. Since your engineers assume the client role from an already-assumed session in the hub, you’re chaining by definition. Plan around the hour rather than fighting it.

An authorization message you can’t read

Some denials come back with a long encoded blob attached. STS will decode it if your identity has permission to, and the decoded version usually names the exact statement that denied you.

aws sts decode-authorization-message --encoded-message <encoded-blob>

Common mistakes

• Naming the account root as the trust principal instead of the specific operations role
• Letting the client pick the external ID, or reusing one value across several clients
• Treating the role session name as an audit trail when it’s a caller-supplied string
• Asking a client for console credentials because a template felt like too much effort
• Granting AdministratorAccess for a migration and never revisiting it
• Building the hub properly and then leaving one legacy IAM user in one account as a fallback
• Assuming that removing someone from your directory also removed the keys they created by hand

Best practices worth the effort

• One identity plane, federated, with MFA enforced at the point the human authenticates
• A unique unguessable external ID per client, generated by you and stored with client metadata
• Source identity required in the trust policy so CloudTrail names a person, not a role
• Roles tiered by purpose: read-only for investigation, change for the engagement, break-glass alarmed
• Client-owned policies, deployed by the client from your template, revocable by the client in one action
• A version-controlled inventory of every account, role, scope and review date
• Access Analyzer external access findings reviewed on a schedule, in every client account

FAQ

Can’t I just ask each client to create an IAM user for me?

You can, and it’ll work on day one. The problem is day four hundred. That user’s access key never expires, it isn’t attached to any individual on your team, and revoking it means someone in the client’s organisation remembering to act. A role costs the client the same five minutes to create and hands them a control they can exercise unilaterally.

Do I need a different external ID for every client?

Yes. A shared external ID across your client base defeats the purpose, because the value’s entire job is to be specific to one trust relationship. Generate a UUID per client, treat it as a secret, and never let the client choose it.

Should I bring client accounts into my own AWS Organization?

Only if you’re also taking on their billing and the commercial relationship supports it. Inside your organisation you get service control policies, resource control policies and centralised root access management, which are real advantages. Outside it, those stay the client’s tools and your job is to recommend them well. Most consulting engagements sit outside.

How do I prove a departed engineer no longer has access?

Show the directory record disabling them, then show that no IAM user or access key in the client account carries their name, then show Access Analyzer external access findings listing only your operations role as an external principal. Three artefacts, all exportable. That’s the evidence pack.

Does MFA carry across the cross-account role assumption?

Don’t assume it does. The reliable place to enforce MFA is where the human authenticates in your identity plane, and the reliable control at the client boundary is the combination of a narrowly scoped principal and the external ID condition. If you want an MFA condition in the client’s trust policy as well, test it end to end against your actual federation path before you rely on it, because the request context differs between direct assumption and chained sessions.

What’s the minimum viable version if I only have three clients?

A hub account with federated engineers, one role per client scoped to a named principal with a unique external ID, and source identity required. Skip the tiering, skip the IP conditions, skip the JIT ambitions. Those three things give you revocation, attribution and a clean answer for the auditor, which is most of the value.

The one thing to take away

Client AWS account access is not really a permissions problem. It’s a revocation and attribution problem wearing a permissions costume. Any design where a credential exists that isn’t tied to a specific human will fail you at exactly the wrong moment, and no amount of policy tuning fixes that, because the gap isn’t in the policy.

Put the humans in one place you control. Put a narrowly scoped, externally identified role in each client account. Require source identity so the log names a person. Everything else on this page is refinement; those three are the structure. And the pleasant side effect is that “here’s how you revoke us, unilaterally, in one action” is a genuinely good thing to be able to say in a sales conversation.

Need a second pair of eyes on your client access model?

This is the kind of work I do for consultancies and managed service providers who’ve grown past the point where a shared vault is survivable. Specifically:

• Reviewing existing cross-account trust policies for over-broad principals and missing external IDs
• Designing a hub account and federation model that scales past a handful of clients without per-client special cases
• Building the CloudFormation or Terraform template clients deploy themselves, so onboarding stops involving credentials
• Turning on source identity end to end, including the identity provider side, so CloudTrail attributes to named engineers
• Producing the offboarding evidence pack clients and auditors ask for, and automating it so it isn’t a scramble
• Narrowing long-standing administrator roles down using service last accessed data, without breaking the engagement

If you want a concrete starting point, send me one trust policy and the AccessDenied message it produces. That’s usually enough to tell where the model is leaking.

The post Twenty Client AWS Accounts and Not One Shared Admin Credential appeared first on John Nessime.

It is a one-line change. It works on the first try. And it is the grant most likely to still be sitting there long after that analytics project shipped, covering a wider set of identities than the person who wrote it ever had in mind.

That asymmetry is the whole problem. A denied cross-account request is a nuisance: someone pings you, you read the policy, you fix it before lunch. A cross-account grant that succeeds, keeps succeeding, and quietly reaches further than intended produces no error, no alert, and no ticket. It surfaces months later during an audit, when someone asks who has been reading that prefix and the honest answer is “anyone the other account decided to let in.”

This post covers AWS cross-account data sharing as a set of failure families rather than a service tour. How the delegation model actually behaves, where encryption keys break or widen access without warning, who ends up owning the bytes after a handoff, why a copy is worse than a reference, and which guardrails catch the mistake you are eventually going to make. Real policy snippets and commands throughout, plus the denials that are hardest to read.

AWS cross-account data sharing is a delegation, not a permission

Start with the mechanic that everything else hangs off. For a request that crosses an account boundary, two separate accounts have to say yes. The resource-based policy in the account that owns the data has to allow the caller, and an identity-based policy in the calling account has to allow the action. Neither side alone is enough.

That is why “I added them to the bucket policy and it still fails” is such a routine ticket. The producer did their half. Nobody attached the matching IAM policy on the consumer side.

Now the part that bites. Look at how most cross-account bucket policies get written:

{
  "Sid": "AllowAnalyticsAccount",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::shared-events-bucket",
    "arn:aws:s3:::shared-events-bucket/exports/*"
  ]
}

Despite how it reads, that :root suffix does not mean the root user. It means the account. You have not granted access to a principal at all. You have delegated the guest list to the administrators of account 111122223333, and they can hand that access to any principal they like, including a role somebody creates next quarter for a purpose you will never hear about.

Sometimes that delegation is exactly what you want, because the consumer team manages its own roles and you do not want to be in the loop for every change. Usually you want the narrower version:

{
  "Sid": "AllowAnalyticsRoleOnly",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/analytics-reader" },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::shared-events-bucket",
    "arn:aws:s3:::shared-events-bucket/exports/*"
  ],
  "Condition": {
    "StringEquals": { "aws:PrincipalOrgID": "o-abc123def4" }
  }
}

Both halves earn their place. The named ARN stops the wrong role in the right account. The organization condition stops the right role name in the wrong account, which matters if the statement is ever copied, templated, or widened by someone in a hurry. One protects against internal drift, the other against the policy escaping your organization entirely.

Be clear-eyed about the trade-off, though. Naming ARNs is precise and brittle: every new consumer role is a policy change, a pull request, and a review. Organization-wide conditions scale beautifully and narrow nothing inside a member account. If your organization contains twelve accounts and forty teams, aws:PrincipalOrgID on its own is a fairly wide door.

The grant that stops working after a rebuild

When you name a role ARN as a principal, IAM stores the role’s internal unique ID behind the scenes. Delete that role and the policy will start displaying an opaque AROA... string where the ARN used to be, because the ARN no longer resolves to anything. Recreate a role with the identical name and the access does not come back, because the new role has a new unique ID.

A terraform destroy followed by terraform apply on the consumer side reproduces this perfectly. If you are staring at a bucket policy that contains a principal starting with AROA, that is your answer. Rewrite the statement with the current ARN.

The encryption boundary nobody tests until it is live

Second failure family: the key. Encrypting shared data is the easy decision. What people miss is that a KMS key is a second, fully independent authorization surface, with its own resource policy, evaluated separately from the bucket.

• Downloading an object encrypted with SSE-KMS requires kms:Decrypt on the key.
• Uploading one requires kms:GenerateDataKey.
• A multipart upload requires both.

The AWS managed key for S3 cannot carry you here, because you cannot edit its key policy. Sharing KMS-encrypted objects across accounts means a customer managed key, with a statement naming the consumer:

{
  "Sid": "AllowAnalyticsRoleToDecrypt",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/analytics-reader" },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*",
  "Condition": {
    "StringEquals": { "kms:ViaService": "s3.us-east-1.amazonaws.com" }
  }
}

The kms:ViaService condition is cheap insurance. It restricts use of the key to requests that arrive through S3 in that region, so a leaked set of consumer credentials cannot call KMS directly and start decrypting ciphertext it obtained some other way. And the two-key rule applies to the key exactly as it does to the bucket: the consumer’s own IAM policy has to allow the KMS actions too.

Now the sharp edge, which is worth reading twice. If a KMS key is specified by alias rather than by fully qualified ARN, the alias is resolved in the requester’s account. In a cross-account write that means data landing in your bucket can end up encrypted under a key belonging to the writer. You own the object and cannot read it, and no error is raised at write time. Use full key ARNs in anything that crosses an account boundary.

Who owns the bytes after the handoff

Third family: ownership, and it depends entirely on which direction the data moves.

New buckets are created with Object Ownership set to Bucket owner enforced. ACLs are off, and the bucket owner owns every object written into the bucket regardless of who wrote it. That default removed the single most confusing trap in S3 sharing.

Buckets created before that default are the ones to check. If a bucket still sits on the Object writer setting, an object uploaded by another account is owned by that account, and you, holding the bucket, may not be able to read it. Nothing fails at write time. It fails later, when a job in your own account tries to read your own bucket and gets a 403.

Check and fix it with the ownership controls API:

aws s3api get-bucket-ownership-controls --bucket shared-events-bucket

aws s3api put-bucket-ownership-controls 
  --bucket shared-events-bucket 
  --ownership-controls '{"Rules":[{"ObjectOwnership":"BucketOwnerEnforced"}]}'

Order matters when you make that change on a live bucket. Once ACLs are disabled, any PUT that still carries an ACL is rejected with AccessControlListNotSupported. Requests that send no ACL, or send bucket-owner-full-control, keep working. So the sequence is:

1. Find every writer that sets an ACL, including old scripts using --acl and any SDK call passing an ACL argument.
2. Update those clients so they stop sending one.
3. Confirm from CloudTrail that ACL-bearing PUTs have stopped.
4. Flip the bucket to Bucket owner enforced.

Given a free choice of design, I would rather the consumer pull from a bucket I own than push into a bucket I do not. Pull keeps ownership, lifecycle rules, encryption configuration, and access logging in one account, under one team’s control. Push scatters them.

Copies you cannot recall

Fourth family, and the one with the longest tail. There is a large difference between granting a reference and shipping a copy. A reference can be revoked on a Tuesday afternoon. A copy is simply gone.

Three reference-shaped patterns are worth knowing before you reach for a bulk export.

S3 Access Points for object-level sharing

Instead of a bucket policy that grows a new statement every time a team asks for something, you write the bucket policy once to delegate to access points, then give each consumer their own access point with its own focused policy. The delegation is expressed with the s3:DataAccessPointAccount condition. Access points can also be owned by the consumer’s account, so they manage their own detail without touching your bucket policy.

The gotcha: when a request goes through an access point, both the access point policy and the underlying bucket policy have to allow it. A 403 here is easy to misdiagnose because the bucket policy looks correct and is correct. The access point is the layer that said no.

Lake Formation for table-level sharing

When the thing being shared is a table rather than a prefix, Lake Formation is the better fit, because it lets you grant on databases, tables, columns, and row filters instead of on object paths. Cross-account grants ride on AWS Resource Access Manager, so the recipient gets an invitation that a data lake administrator has to accept.

aws ram get-resource-share-invitations

aws ram accept-resource-share-invitation 
  --resource-share-invitation-arn arn:aws:ram:us-east-1:111122223333:resource-share-invitation/EXAMPLE

After the invitation is accepted, the shared database and tables are visible but still not queryable from Athena or Redshift Spectrum until someone creates a resource link, which behaves roughly like a symlink in the recipient’s Data Catalog. Missing resource links account for most of the “the share went through but I cannot see the table” messages.

One setting to check on the producer side is the cross-account data sharing version. Newer versions collapse many grants into far fewer RAM resource shares, which matters once you are sharing at any scale, and they allow granting directly to IAM principals in another account rather than only at account level. Read the current version with:

aws lakeformation get-data-lake-settings

When a copy really is the answer

Sometimes the partner genuinely needs the data on their own infrastructure: a warehouse you have no reach into, a VPS at a provider like Contabo or InterServer, or in the worst case an analyst’s laptop. That is a legitimate outcome, but be honest with yourself and with them that your controls stop at the handoff.

Practical hygiene for that case: scope the extract to the columns actually needed rather than the whole table, put an expiry date in the agreement rather than in your head, and write down what happens to the copy when the engagement ends. If the copy lives on machines you or the client control, verified erasure with something like O&O SafeErase reads a great deal better in an audit than “we deleted the folder.” And if the recipient has no AWS account at all, a time-limited presigned URL is usually a better answer than minting an IAM user with long-lived keys.

The guardrail that catches the mistake you will make

Everything above describes what a careful engineer does per grant. Guardrails are what happens when somebody is not careful, which over a long enough window includes all of us.

Resource control policies are the strongest lever if you run AWS Organizations. An RCP sets a ceiling on what any resource-based policy inside the organization can grant. The canonical statement is “no principal outside my organization may access S3 buckets in my accounts, regardless of what an individual bucket policy says.” That inverts the risk model: a careless bucket policy is no longer an exposure, it is just an ineffective statement.

Two limits are worth stating plainly. RCPs cover a specific set of services rather than everything in AWS, so check the current supported list rather than assuming coverage. And they constrain your resources, not a partner’s behavior once data has legitimately left. An RCP is a wall around your side of the fence.

Service control policies do the mirror-image job on the consumer side, limiting which external accounts your own principals are allowed to assume roles into. Account-level S3 Block Public Access remains a sensible floor underneath both.

For sharing with a third party rather than an internal account, the external ID is not optional in my book:

{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::444455556666:role/vendor-connector" },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringEquals": { "sts:ExternalId": "a-value-the-vendor-generated-for-you" }
  }
}

The external ID exists because of the confused deputy problem. A vendor holds credentials that can assume roles across many customers, and without a per-customer secret in the condition, customer A can trick the vendor into pointing that access at customer B. If a vendor asks you to create a role for them with no external ID and no conditions at all, that is worth a conversation before it is worth a role.

Verifying that the grant is what you think it is

Reading policies is not the same as knowing what they permit. IAM Access Analyzer answers the question “what is currently reachable from outside my zone of trust” using automated reasoning rather than pattern matching, which is why it catches combinations a human reviewer skims past.

aws accessanalyzer create-analyzer 
  --analyzer-name external-access 
  --type ACCOUNT

aws accessanalyzer list-findings 
  --analyzer-arn arn:aws:access-analyzer:us-east-1:999988887777:analyzer/external-access

Scope the zone of trust to your organization where you have one, then archive the findings that represent intentional shares, so that a new entry in the list actually means something.

The same service exposes custom policy checks you can run in a pipeline before a policy change merges, including checks for public access, for specific actions being granted, and for whether an updated policy grants more than the one it replaces. One thing to understand about them: they evaluate the document you hand them and nothing else. They do not read your account state, so a public-access check can fail on a policy in an account that has Block Public Access enabled. For a merge gate that is correct behavior, because the policy should not be relying on a distant safety net.

Finally, CloudTrail tells you who actually used a grant, which is the input to the only cleanup that ever really works: delete the statements nobody has exercised. Routing that history into CloudWatch, or into a Grafana Cloud dashboard alongside the rest of your platform telemetry, turns “is this still needed” from an argument into a query.

Troubleshooting the denials that are hard to read

• You can list the bucket but every GET returns 403. That pattern points at KMS, not S3. Listing does not touch the key; reading does. Check kms:Decrypt on both the key policy and the consumer’s identity policy.
• 403 through an access point while the bucket policy is clearly correct. The access point has its own policy and it also has to allow the calling principal. Read that one next.
• AssumeRole is denied although the trust policy names the caller. Cross-account needs both halves: the caller’s identity policy must permit sts:AssumeRole against that role ARN. Then check the external ID value and any SCP or permissions boundary in the calling account.
• A principal in the policy displays as AROA followed by random characters. The referenced role was deleted. Recreating it under the same name will not restore access.
• A Lake Formation share appears in the catalog but queries fail. Usually a missing resource link in the recipient account, or a RAM invitation that was never accepted.
• Access worked yesterday and stopped today with nothing changed locally. Look at the organization layer before re-reading the bucket policy. A newly attached RCP or SCP denies in a way that no amount of resource-policy editing will fix.

Common mistakes

• Granting to an account principal when a role ARN would have done, then never revisiting it.
• Treating aws:PrincipalOrgID as least privilege. It narrows the organization, not the account.
• Sharing a bucket and forgetting the key, then debugging S3 for an hour over a KMS denial.
• Referring to KMS keys by alias in cross-account configuration instead of by full ARN.
• Letting a partner push into your bucket without confirming the ownership setting first.
• Exporting a full table when the consumer needed four columns, because the export was easier to write.
• Creating a vendor role with no external ID and no conditions because the vendor’s setup guide said to.
• Granting access with no expiry, no owner, and no note in the policy Sid explaining why it exists.

Best practices worth the effort

• Give every cross-account grant a descriptive Sid. It is the only free documentation slot IAM offers, and future-you will read it.
• Prefer one dedicated consumer role per relationship over a shared role used by several integrations. Revocation becomes a single delete.
• Pair a specific principal with a broad condition, rather than relying on either alone.
• Put policy checks in CI so a widened grant fails a pull request instead of a quarterly review.
• Keep an inventory of active shares with an owner and a review date, even if it is a table in a repository README.
• Enable an external access analyzer scoped to your organization and treat new findings as work, not noise.
• Watch cross-region and internet data transfer when a consumer sits somewhere unexpected. Cost visibility tooling such as Vantage or CloudZero will attribute it faster than reading a bill.

Frequently asked questions

What is the safest way to share an S3 bucket with another AWS account?

Name a specific role ARN in the bucket policy, add an organization or account condition alongside it, scope the resource to a prefix rather than the whole bucket, and make sure the consumer’s identity policy is equally narrow. If the bucket is encrypted with a customer managed key, grant on the key as well. If the list of consumers is going to grow, move to access points rather than growing the bucket policy.

Do I need both a bucket policy and an IAM policy?

For cross-account access, yes. The resource-based policy in the owning account and the identity-based policy in the calling account both have to allow the action. That is the single most common reason a correct-looking bucket policy still returns Access Denied.

Why do I get Access Denied when the bucket policy clearly allows the other account?

Work through it in this order: the consumer’s identity policy, then the KMS key policy if the objects are encrypted with a customer managed key, then any access point policy in the path, then organization-level SCPs and RCPs. Denials from the organization layer are the ones people waste the most time on, because nothing in either account’s policies looks wrong.

Can I share data with an account outside my AWS Organization?

You can, and organization conditions will not help you there. Use a named role ARN, an external ID on any assumed role, a tightly scoped resource, and a defined end date. Also confirm whether an RCP in your organization already blocks external principals for that service, because that is a common reason a partner grant refuses to work no matter how carefully it is written.

Should I use S3 Access Points or Lake Formation?

It depends on the unit of sharing. If the consumer wants objects under a prefix, access points are simpler and cheaper to operate. If the consumer wants a table, and especially if you need column or row filtering, Lake Formation is the right tool and access points would force you to fake fine-grained control with prefix layout. Plenty of environments run both.

How do I revoke cross-account access cleanly?

Remove the statement from the resource policy, remove the KMS key grant, and check for any access point, RAM share, or resource link that also carries permission. Then confirm with CloudTrail that the principal has stopped appearing. Revocation feels done long before it is done, which is exactly why an inventory of active shares pays for itself.

Does cross-account access change anything about cost?

The request charges land on the bucket owner by default, and data transfer depends on whether the consumer is in the same region. A consumer that quietly starts scanning a large prefix from another region is a billing event as much as a security one, which is a good argument for reviewing access patterns and spend together rather than in separate meetings.

The one thing to remember

Getting AWS cross-account data sharing right is less about knowing which service to reach for and more about noticing which grants are quietly larger than they read. An account principal delegates your guest list. An organization condition narrows the wrong dimension. A copy leaves your perimeter permanently and no policy edit brings it back.

Grant references rather than copies, name principals rather than accounts, remember that the key is a second door, and put something in place that watches the perimeter for you. The cross-account grant worth worrying about is never the one that failed. It is the one that worked, and kept working, long after anybody was still looking at it.

Need a second pair of eyes on your cross-account setup?

Most of the cross-account problems I see are not exotic. They are a policy written in a hurry two years ago that nobody has read since. If that sounds familiar, here is where I can help:

• Auditing existing bucket, key, and role trust policies to find grants that are wider than intended, and rewriting them to named principals with conditions.
• Designing a producer and consumer pattern for a specific data handoff, including whether access points, Lake Formation, or a plain scoped role is the right shape.
• Setting up IAM Access Analyzer with a sensible zone of trust and archive rules, so the findings list stays worth reading.
• Adding policy checks to your pipeline so a widened grant blocks a merge rather than surviving to production.
• Building an organization-level perimeter with resource control policies and service control policies without breaking the integrations you already depend on.
• Untangling stubborn cross-account denials, including the KMS, access point, and organization-layer cases that hide well.

Send me a bucket policy, a trust policy, or the CloudTrail entry for the denial you cannot explain, and I will tell you what it actually permits.

The post AWS Cross-Account Data Sharing Done Safely: Share Access, Not Copies appeared first on John Nessime.

That is the failure mode that actually bites when you build a medical lab data pipeline on AWS. Not the bucket you thought about. The seven places PHI ended up because a payload got logged, an error got queued, or a filename contained a medical record number.

This post walks through the design boundary by boundary: where lab data crosses from one trust zone into another, what leaks at each crossing, and which control closes it. It is aimed at engineers who have been handed an HL7 feed and a compliance checklist and told to make them meet. I am an engineer and not your compliance counsel, so treat the regulatory references here as pointers to read the primary text yourself.

Why laboratory data is a harder pipeline than it looks

Lab data has three properties that most ETL work does not.

First, results get corrected. A preliminary result goes out, the analyzer is recalibrated, and a corrected result follows hours or days later. Your pipeline is not append-only in the way a clickstream is. It has to carry amendment semantics all the way through, and an analytics table that silently keeps the first value is worse than no table at all.

Second, the wire format is old and the transport is older. Most laboratory information systems still emit HL7 version 2 messages, pipe-delimited segments carried over the Minimum Lower Layer Protocol, which is framed bytes over a raw TCP socket. AWS documents MLLP as the common transport standard for HL7 v2 interoperability in its Healthcare Industry Lens. It has no built-in authentication and no built-in encryption. Whatever security it has, you bolt on around it.

Third, the identifiers are load-bearing. Accession numbers, specimen IDs and medical record numbers are how the lab reconciles anything, so PHI is not a column you can drop. It is structural, and it will try to escape into every metadata surface you own.

Boundary one: getting messages out of the lab

The instrument or LIS sits on a network you probably do not control, run by a vendor who may not let you touch the configuration. That constrains the design more than anything on the AWS side.

You get three realistic shapes, and the LIS vendor usually picks for you.

• Live MLLP over a private link. An interface engine in the lab holds the TCP session and forwards over a Site-to-Site VPN or Direct Connect into a private subnet. Lowest latency, highest operational burden, because a dropped socket is a clinical incident and not a retry.
• Batch file drop over SFTP. The LIS writes flat files on a schedule and something picks them up. Far easier to reason about, far easier to audit, and acceptable whenever the downstream use is analytics rather than result delivery.
• An outbound API push. Newer systems will POST to an endpoint you expose. Best case, because you control authentication and you get a response code back, but rarest in practice.

Whichever shape you get, the same rule applies at this boundary: terminate the untrusted protocol as early as possible and convert to something you can authorize. A raw MLLP socket carries no identity. The moment those bytes land, wrap them in a message you can attribute, sign, and trace.

Two things go wrong here. First, people treat a network tunnel as authentication. A VPN tells you the packets came from the lab’s network. It does not tell you which system sent them, or that the sender was authorized to send that patient’s results. Second, people assume consumer VPN products fill the gap. Tools like NordVPN or Surfshark are fine for your own admin laptop on untrusted wifi, but they carry no business associate agreement and they are not a substitute for AWS Site-to-Site VPN or AWS Client VPN on a clinical path.

One piece of housekeeping that is cheap to do and expensive to skip: the business associate addendum. AWS makes this self-service through AWS Artifact, and it can be accepted across an entire organization. Only services on the current HIPAA Eligible Services Reference may hold PHI, and that list changes, so check it against the services you actually chose rather than your memory of it.

Boundary two: the landing zone

Raw messages land in S3. This part is well documented, so I will focus on the parts people configure once and never verify.

Make the bucket policy do the enforcing

Default bucket encryption means objects get encrypted. It does not mean a caller cannot write an object with different settings, and it does not stop a plaintext HTTP request. Two explicit denials cover the gap. The first protects the transport, the second protects the storage:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::lab-raw-inbound",
        "arn:aws:s3:::lab-raw-inbound/*"
      ],
      "Condition": {
        "Bool": { "aws:SecureTransport": "false" }
      }
    },
    {
      "Sid": "DenyWrongKey",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::lab-raw-inbound/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption-aws-kms-key-id":
            "arn:aws:kms:REGION:ACCOUNT:key/KEY-ID"
        }
      }
    }
  ]
}

Use a customer managed KMS key, not the AWS managed one. The reason is not stronger cryptography, it is the key policy. A customer managed key gives you a second, independent authorization surface: even a principal with broad S3 permissions cannot read the object if the key policy does not grant them decrypt. That separation is what makes a blast radius argument credible to an assessor, and it is what makes crypto shredding possible later.

Your object keys are metadata, and metadata leaks

This is the one I would put on a poster. Object contents are encrypted. Object keys are not. They show up in CloudTrail data events, S3 server access logs, S3 Inventory reports, bucket listings, Lambda event payloads, and every console screenshot anyone pastes into a ticket.

# Leaks PHI into six places that are not the bucket
raw/2024/ORU_MRN4471902_SMITH_JOHN_A1C.hl7

# Same object, no identifiers in the key path
raw/dt=<partition>/src=lis01/msg=01HXYZ...ULID.hl7

Use an opaque, sortable identifier and keep the mapping from identifier to patient inside an encrypted store you actually control. Partition on ingest date and source system, never on anything derived from the patient. This one decision removes more PHI surface than any other single change in the pipeline.

Immutability, and the trap inside it

Clinical laboratories in the United States operating under CLIA have record retention obligations set out in 42 CFR 493.1105. The floor for test reports is at least two years after the date of reporting, with pathology test reports at ten years, and several state requirements sit above the federal floor. Retention is therefore not something you invent; go and read the section, then check your state.

S3 Object Lock gives you write-once-read-many enforcement, and it is the right tool. But understand the two modes before you commit, because one of them is genuinely irreversible.

• Governance mode blocks deletion for most principals, but anyone holding s3:BypassGovernanceRetention can override it by sending the x-amz-bypass-governance-retention:true header. Note the S3 console includes that header by default.
• Compliance mode cannot be overridden by anyone, including the account root. Per AWS documentation, the only way to delete an object under compliance retention before its date expires is to delete the AWS account.

A typo in a retention date under compliance mode is permanent, and you pay to store the mistake for its full term. Object Lock also has to be enabled at bucket creation, alongside versioning. Start in governance mode, prove the retention values against real traffic, then move to compliance mode deliberately rather than as a default.

# Indefinite hold for a specific object version, e.g. under litigation
aws s3api put-object-legal-hold 
  --bucket lab-raw-inbound 
  --key raw/dt=.../msg=01HXYZ.hl7 
  --legal-hold Status=ON

Boundary three: where PHI escapes into your observability

Here is the section that justifies the post. Everything above is on the compliance checklist. This is the part that is invisible until an audit or an incident makes it visible.

Your operational plane is a PHI store you did not declare. Specifically:

• Application logs. One print(record) left in a parser during debugging writes patient demographics into CloudWatch Logs, where the default is often no expiry at all.
• Exception traces. Unhandled parse errors frequently include the offending segment in the message. That segment is the PHI.
• Step Functions execution history. If you pass message bodies between states as payloads, that content is retained in the execution record and readable by anyone who can view executions.
• Dead letter queues. The whole point of a DLQ is to keep the failed message. A DLQ full of unparseable HL7 is a PHI archive with a different access policy than the bucket you designed so carefully.
• Third-party monitoring. Anything shipped to an external platform leaves your boundary. Grafana Cloud, Datadog and equivalents are perfectly reasonable choices for infrastructure telemetry, but a log line containing PHI going to a vendor without an executed BAA is a disclosure, not a metric.
• Notifications. An SNS topic that emails an on-call engineer “failed to process record for J. Smith, DOB …” has just sent PHI to a mail server nobody reviewed.

The pattern that fixes it: pass pointers, not payloads

Adopt a single rule and enforce it in review: PHI moves by reference between components, never by value.

1. The ingest function writes the raw message to the encrypted bucket and emits only an opaque object key plus a message ID.
2. Every downstream state, queue and event carries that pointer. Nothing carries the record body.
3. Error handling logs the pointer and a failure class, never the content that failed.
4. A DLQ therefore contains pointers to failures, and reprocessing means re-reading from the bucket under the same key policy as everything else.

The cost of this pattern is a slightly noisier debugging experience. You cannot read the failing record straight out of the queue. You have to go and fetch it, with credentials, and that fetch is logged. That inconvenience is the control working.

Then close the log surface itself

Log groups created implicitly by Lambda and other services often retain data indefinitely unless you set a policy. Find the ones nobody configured:

# List log groups with no retention configured
aws logs describe-log-groups 
  --query 'logGroups[?retentionInDays==null].logGroupName' 
  --output text

# Set an explicit retention on one of them
aws logs put-retention-policy 
  --log-group-name /aws/lambda/hl7-ingest 
  --retention-in-days 30

Run the first command as a scheduled check, not a one-off. New functions create new log groups, and the default comes back every time someone deploys.

Amazon Macie is worth pointing at your buckets as a detective backstop. It will not stop a leak, but it will tell you when something started landing where it should not.

Boundary four: from identified data to analytics

At some point someone wants to query the data. This is the crossing where teams are most likely to convince themselves they have done something they have not.

Amazon Comprehend Medical will detect PHI entities in unstructured clinical text through its DetectPHI operation, returning entity types such as name, address, identifier and date, each with a confidence score. It is a genuinely useful tool for narrative fields: pathology comments, specimen notes, microbiology free text.

aws comprehendmedical detect-phi 
  --text "Specimen received from the referring clinic on the stated date."

Now the part the marketing pages underplay, straight from the AWS developer guide: the entities Comprehend Medical detects do not map one to one to the identifier list specified by the Safe Harbor method, and AWS explicitly recommends additional human review or other methods to confirm accuracy for compliance use cases.

Read that as: PHI detection is not de-identification. Under the HIPAA Privacy Rule, de-identification has two defined routes, Safe Harbor and Expert Determination. An NLP model with confidence scores is an input to either, not a substitute for either. If your analytics tier is meant to hold de-identified data, someone has to own that determination, and it will not be your pipeline.

What works structurally:

• Put the analytics tier in a separate AWS account with its own KMS key. Not a separate bucket. A separate account, so the blast radius argument is enforced by IAM boundaries rather than by convention.
• Make the flow one directional. The de-identification job reads from the identified side and writes to the analytics side. Nothing in the analytics account holds permissions pointing back.
• Replace identifiers with pseudonyms rather than deleting them, keeping the crosswalk in the identified account. This preserves your ability to join across results and to re-identify under a documented process, which the research use case usually needs.
• Watch quasi-identifiers. Rare test panels, unusual reference ranges and precise timestamps re-identify people in small populations even after the obvious fields are gone. This is a statistics problem, not an IAM problem.
• Use Lake Formation or equivalent for column and row filtering if analysts need partial access, so the restriction lives with the catalog rather than in each query tool.

Boundary five: sending results back out

Delivery is where clinical correctness and security pull against each other, and security usually loses quietly.

The correction problem is the sharp edge. HL7 result messages carry a status that distinguishes preliminary, final and corrected results. If your delivery layer is idempotent on message ID alone, a corrected result with a new ID for the same observation will happily land beside the original instead of superseding it. Two conflicting values, both marked delivered, nobody alerted.

Build the idempotency key from the identity of the observation, which typically means the placer or filler order identifier combined with the specific observation identifier, and carry the result status as a first-class field so a correction overwrites rather than appends. Keep every version, expose the current one.

For the outbound path itself:

• Presigned URLs are convenient for report PDFs and they are also a bearer credential. Anyone holding the link has the document. Keep expiry short and treat generation as an auditable event.
• If you expose a provider-facing portal, put a WAF in front of it. AWS WAF and Cloudflare are both credible here; the deciding factor is usually where the rest of your edge already lives, not the rule engines.
• Never email a result. Email a notification that a result is available, behind authentication. The distinction sounds pedantic right up until someone forwards a mailbox.

Boundary six: deletion, and the conflict nobody plans for

Two obligations point in opposite directions. Retention rules say keep the record. Privacy rights and internal policy say be able to remove data. If you set compliance-mode Object Lock across everything, you have chosen one side without noticing.

Resolve it by classifying before you lock. Records that fall under laboratory retention obligations go into the immutable tier with a retention period derived from the applicable rule. Derived artifacts, caches, intermediate parquet, analytics extracts and enrichment outputs do not belong there. They belong in buckets with lifecycle rules, and they are what you actually delete.

Crypto shredding covers the middle ground: if a data set is encrypted under a key used for nothing else, scheduling that key for deletion renders the ciphertext unrecoverable without touching the objects. It is clean, it is verifiable, and it only works if you planned the key granularity up front. Retrofitting per-tenant or per-cohort keys onto a pipeline that used one key for everything is a full re-encryption exercise.

Do not forget the endpoints. Instrument workstations, the interface engine box, and the laptop somebody used to test the parser all accumulate copies. Cloud controls do nothing for physical media, and secure erase tooling such as O&O SafeErase exists for exactly that step in a decommissioning runbook.

Troubleshooting a medical lab data pipeline on AWS

Messages arrive but nothing appears downstream

Check the DLQ first, then check whether the DLQ itself has a consumer. A surprising number of pipelines have a correctly configured dead letter queue that nobody monitors, so failures accumulate silently and the only symptom is a gap in the data. Alarm on DLQ depth greater than zero, not on a threshold.

Access denied from a Lambda that has the right IAM policy

With a customer managed KMS key, IAM permission on S3 is only half the grant. The function’s role also needs decrypt permission on the key, and the key policy has to allow it. If you are crossing accounts, both the key policy and the role policy must permit the action. This is the intended behavior of the two-lock design, and it is the single most common support question on any pipeline built this way.

Deletion fails on an object you are certain you should be able to delete

Check for a legal hold before you check retention. A legal hold has no expiry and is independent of the retention period, so an object can be past its retention date and still undeletable. Also confirm which mode the retention uses, because governance and compliance produce the same error to a caller without bypass permission.

Counts do not reconcile with the LIS

Compare on observations, not messages. One message can carry several results, corrections create additional messages for the same observation, and a naive message count will diverge from the LIS by exactly the amount that matters clinically. Reconcile daily and alert on drift rather than investigating at quarter end.

Timestamps drift between systems

HL7 v2 timestamps do not always carry a timezone offset, and analyzers are often set to local time with no daylight saving handling. Capture the offset at ingest from the source system configuration and store everything in UTC with the original string preserved. Recovering an ambiguous timestamp after the fact is unpleasant, and around a daylight saving transition it may not be possible at all.

Common mistakes

• Treating “HIPAA eligible” as “compliant.” Eligibility means AWS will cover the service under the BAA. Configuration is still entirely yours.
• Putting identifiers in object keys, queue names, or log group names. Encrypted contents, plaintext metadata.
• Enabling compliance-mode Object Lock before validating retention values. There is no undo, and no support ticket that fixes it.
• Assuming automated PHI detection equals de-identification. AWS itself says the entity list does not map one to one to Safe Harbor identifiers.
• Running the analytics tier in the same account as identified data. A bucket boundary is a policy away from collapsing. An account boundary is not.
• Shipping application logs to an external platform without checking what the log lines contain. Test it by grepping your own log group for a known test patient name.
• Ignoring corrected results until an analyst notices. Amendment handling is a day-one requirement, not a phase two feature.

Best practices worth the effort

• Write the data flow diagram first and mark every trust boundary crossing. Every arrow that crosses one needs a named control. This exercise finds more problems than any scanner.
• Encode the eligible service list as policy. Service control policies at the organization level stop someone from putting PHI into a service you never assessed.
• Keep VPC endpoints on the PHI path. Interface and gateway endpoints keep traffic to AWS services off the public internet and give you an endpoint policy as an extra choke point. Check which of your chosen services offer one.
• Build with synthetic HL7 from day one. Nobody should need production PHI to develop a parser. A cheap VPS from Contabo or InterServer is fine for a synthetic-data development box, and keeping that environment entirely outside the PHI boundary is the point.
• Alarm on absence. A feed that stops is more dangerous than a feed that errors, because errors are loud. Alert when expected message volume for a source drops below its floor.
• Test restores, not backups. Restore into an isolated account, confirm the KMS grants work there, and document how long it took.
• Review CloudTrail for data events on the PHI buckets. Object-level logging costs money and is the only record of who read what.

FAQ

Do I need a BAA with AWS before I start building?

Before PHI touches the environment, yes. You can architect and test with synthetic data first. AWS provides the business associate addendum through AWS Artifact as a self-service acceptance that can be applied across an organization, and only services listed on the HIPAA Eligible Services Reference may process, store or transmit PHI.

Should I use AWS HealthLake or build on S3, Glue and Athena?

HealthLake is a managed FHIR datastore with FHIR APIs, medical NLP and query built in, and it is the shorter path if your consumers speak FHIR. It does not natively ingest HL7 v2; AWS points to partner tooling or your own transformation for non-FHIR input. If your consumers are analysts with SQL and the data is structured lab results rather than clinical documents, an S3 and Athena lakehouse is simpler and easier to keep inside a boundary you control. Decide on the consumer, not the source.

How long do I have to keep laboratory records?

For CLIA-regulated laboratories in the United States, 42 CFR 493.1105 sets the federal floor, including at least two years for test reports after the date of reporting and at least ten years for pathology test reports. State law frequently exceeds these minimums, so the applicable period is whichever is longer for your jurisdiction. Read the regulation and confirm with your compliance lead before you encode a number into a retention policy.

Can Amazon Comprehend Medical de-identify data for me?

It can detect PHI entities and give you confidence scores, which is a strong starting point for redaction workflows. It does not by itself satisfy HIPAA’s de-identification standard. AWS documents that the detected entities do not map one to one to the Safe Harbor identifier list and recommends human review for compliance use cases. Treat it as a detector feeding a documented Safe Harbor or Expert Determination process.

Is S3 Object Lock compliance mode required for lab records?

Not inherently. The argument for compliance mode is a threat model in which a compromised or malicious administrator could remove retention, not a regulation that names the mode. Governance mode plus tightly controlled bypass permissions and strong audit logging is a defensible position for many laboratories. Choose deliberately and write down the reasoning, because compliance mode commits you to the storage cost for the full term with no exit.

Where does PHI most often leak in a pipeline that looks correctly configured?

Logs, dead letter queues, workflow execution histories, notification messages and object key names. All five are metadata surfaces that sit outside the storage layer everyone reviews. Grep your own log groups for a known test patient identifier and you will find out quickly whether your pipeline has the problem.

Do I need a dedicated AWS account just for the lab pipeline?

Separate accounts for identified and de-identified data are worth it even at small scale, because the boundary is then enforced by IAM rather than by naming conventions. Whether the PHI workload also needs isolating from your other production workloads depends on who holds admin access there. If the answer is “broadly the same people,” separate it.

The one thing to take away

A secure medical lab data pipeline on AWS is not defined by the encryption on its storage layer. Encryption is the easy part, and it is the part everyone gets right. What separates a design that survives an audit from one that does not is whether PHI can travel by value through your operational plane: into logs, into queue bodies, into workflow histories, into filenames, into alert emails.

Move PHI by reference, keep identifiers out of every metadata surface, and put an account boundary between identified and de-identified data. Do those three things and the rest of the compliance work becomes documentation rather than redesign.

Work with me on healthcare data pipelines

I design and review AWS data platforms for regulated workloads, and lab data is a specific enough problem that generic cloud advice tends to miss it. Things I can help with:

• Mapping every trust boundary in an existing pipeline and finding the PHI surfaces nobody documented, including logs, queues and object key schemes.
• Designing HL7 v2 or file-drop ingestion into S3 with KMS key separation, VPC endpoints and bucket policies that enforce rather than suggest.
• Building the identified and de-identified account split, including the one-way transformation job and the pseudonym crosswalk.
• Getting amendment and correction handling right so corrected results supersede rather than duplicate, all the way to the analytics tables.
• Retention and deletion design: Object Lock modes, lifecycle policies, crypto shredding key granularity, and the runbook that ties them together.
• Observability that is useful without being a disclosure risk, including CloudWatch retention hygiene and safe alerting patterns.

If you have an architecture diagram, a Terraform plan, or a redacted sample of the messages you are receiving, send it over and I will tell you what I would change first.

The post Where PHI Escapes: Building a Secure Medical Lab Data Pipeline on AWS appeared first on John Nessime.