That is the failure mode worth internalizing before anything else: Amazon Macie PII detection reports on what it managed to read, not on what is there. Objects it never opened produce no findings, and no findings looks identical to a clean result. The gap between “we scanned this” and “we found nothing” is where most Macie deployments quietly fail their first audit.

This post covers where that gap comes from and how to close it: the eligibility chain every object goes through, the four distinct reasons an object gets skipped, how to tune identifiers so the findings are worth reading, what actually drives the bill, and how to route results somewhere a human will act on them.

The eligibility chain behind Amazon Macie PII detection

Every object passes the same gates before a byte gets inspected. The order matters, because each gate fails differently and needs a different fix.

1. Is it in a general purpose S3 bucket? Directory buckets are out of scope entirely.
2. Is the storage class supported? Standard, Standard-IA, One Zone-IA, Intelligent-Tiering, Glacier Instant Retrieval and Reduced Redundancy are in. Glacier Deep Archive and S3 Express One Zone are not.
3. Does the object key carry a recognized file extension? This is a string check on the name, not a content sniff.
4. Can Macie retrieve and decrypt it, given the bucket policy, object ACL and encryption key?
5. Does the content parse, and stay inside the per-file quotas?
6. Do the active data identifiers match anything in it?

Only the last gate produces a finding. Everything before it produces silence. Macie does record why each object was skipped, but that lives in coverage data and object samples, not on the findings page most people look at.

One detail explains a lot of confusion: when you first enable automated sensitive data discovery, every bucket gets a sensitivity score of 50 and the label Not yet analyzed. A bucket whose permissions block Macie stays there permanently. It never turns red. It sits mid-list, looking unremarkable next to buckets that genuinely were analyzed.

Gap one: objects Macie never opened

Macie calls these unclassifiable. They fail the storage class or extension check, so no retrieval is attempted. This is the largest source of false confidence I’ve seen, and the cheapest to fix.

Missing file extensions

Classifiability comes from the file name extension. A file full of customer records named part-00003-a4f9 is invisible. The same bytes named part-00003-a4f9.json get inspected and produce findings.

This bites hardest on streaming ingestion. Amazon Data Firehose writes to S3 without appending an extension unless compression, format conversion, or the explicit file extension setting adds one. An uncompressed JSON stream lands as a tree of extensionless objects. Athena and Glue read them fine, because they infer format from the table definition. Macie skips every one.

Fix it upstream: set the S3 file extension on the delivery configuration, or enable GZIP compression or Parquet conversion, both of which append a recognized extension. If you can’t change the producer, the fallback is a copy-and-rename step into a scan prefix, which costs you storage plus a pipeline to maintain. Fixing the producer is almost always cheaper.

Unsupported storage classes

A lifecycle rule transitioning old exports to Glacier Deep Archive quietly removes them from scope. That may be the right storage decision, but it needs to be a conscious one, because archived customer data is still customer data when a regulator asks.

Macie exposes these counts per bucket, split by cause. This is the first query I run against a new account:

# Buckets where Macie can't classify objects, broken down by cause
aws macie2 describe-buckets 
  --query 'buckets[?unclassifiableObjectCount.total > `0`].{
      bucket: bucketName,
      classifiable: classifiableObjectCount,
      skipped_total: unclassifiableObjectCount.total,
      skipped_extension: unclassifiableObjectCount.fileType,
      skipped_storage_class: unclassifiableObjectCount.storageClass
    }' 
  --output table

Read the ratio, not the raw number. Forty thousand classifiable objects and twelve skipped is fine. Three classifiable and four hundred thousand skipped is a reporting artifact pretending to be a scan result, and that bucket’s sensitivity score means nothing.

Gap two: objects it opened and couldn’t read

These are classification errors rather than eligibility failures. Macie selected the object, tried to fetch it, and was refused. Three causes, and they need three different people to fix them.

• Customer-provided keys (SSE-C). Macie cannot supply the key material, so it cannot retrieve the object. No permissions fix exists. Re-encrypt with S3 managed or KMS keys if you want coverage.
• KMS key policy. For customer managed keys, the key policy must allow the Macie service-linked role, AWSServiceRoleForAmazonMacie, to decrypt. Cross-account buckets need the key owner to grant it, not the bucket owner. AWS publishes a permission analyzer script in the aws-samples/amazon-macie-scripts repository that enumerates every key Macie needs and generates the CLI commands to fix them.
• Restrictive bucket policies. An explicit Deny conditioned on source IP or VPC endpoint blocks Macie along with everything else. The working pattern excludes the service-linked role ARN from the deny using the aws:PrincipalArn condition key.

To see which objects Macie actually touched in a bucket, pull the object samples. It’s the closest thing to a scan log you get:

# Objects automated discovery selected, with per-object status
aws macie2 list-resource-profile-artifacts 
  --resource-arn arn:aws:s3:::your-bucket-name

Anything with a status of SKIPPED is a lead. Take the key, check its encryption settings in S3, and you usually have your answer inside a minute.

Gap three: it read the file and nothing matched

Macie ships managed data identifiers covering common PII, financial data and credentials across many countries. Automated discovery uses a recommended subset by default rather than all of them. Sensible, but it is a default, and it will miss things that matter to you.

Add the managed identifiers for your actual jurisdictions. If you hold records for customers in a country whose national ID identifier isn’t in the recommended set, you’re scanning for the wrong things with perfect efficiency. Pull the current list with aws macie2 list-managed-data-identifiers and compare it against where your customers live.

Write custom identifiers for what only you know is sensitive. Internal account numbers, case references, employee IDs. A custom identifier is a regex plus optional keywords and a proximity rule, and the keywords are what save you. A bare [0-9]{8} matches timestamps, order totals and row counts, and you drown. The same regex with keywords and a match distance only fires when the number sits near a word that gives it meaning.

Test the pattern before it goes near a job. This runs the criteria against sample text and returns match counts without creating anything:

# Dry-run a custom identifier before creating it
aws macie2 test-custom-data-identifier 
  --regex 'ACC-[0-9]{4}-[0-9]{4}' 
  --keywords account customer acct 
  --maximum-match-distance 30 
  --sample-text 'customer account ACC-4821-9930 updated; total 1234-5678'

Run it against real samples, including the awkward ones, then against a file you know is clean and confirm zero. A pattern you only tested on positives is a pattern you haven’t tested.

The mirror image is allow lists, which tell Macie to ignore specific text or patterns: your support inbox address, your published press contact, the seeded test records in everyone’s staging fixtures. Without them the same non-issues appear in every scan until the team stops reading findings. Allow lists accept predefined text or regex, and the per-account and per-job quotas are tight enough that you should curate rather than dump.

Gap four: quota truncation inside large files

Subtle, because the object is analyzed. Just not completely, and a partial result reports like a full one.

• Per-file size quotas differ by format. Exceed the quota for a type and Macie analyzes none of that file, not part of it.
• Archives have separate ceilings for nesting depth, extracted bytes and file count. If the metadata shows a breach up front, nothing is extracted. Cross a limit partway through and Macie stops, reporting only what it processed.
• JSON and JSON Lines have a nested depth limit. Past it, the file is not analyzed at all.
• Full names and mailing addresses cap out per file. After the cap Macie stops counting and stops reporting locations for that type, so the occurrence count is a floor, never a total.

Findings carry a status reason naming the limit that was hit, with values like FILE_EXCEEDS_SIZE_LIMIT, ARCHIVE_NESTING_LEVEL_OVER_LIMIT and JSON_NESTING_LEVEL_OVER_LIMIT. Filter on those before declaring a data lake of large compressed exports clean. Check current quota values in the Macie documentation rather than memorizing them, since they differ per format.

Automated discovery or a discovery job?

Automated sensitive data discovery evaluates your inventory daily and samples representative objects across as many buckets as it can, grouping by metadata like prefix, extension and last-modified date. It is breadth-first and it does not re-analyze unchanged objects. Its job is to tell you which buckets deserve attention. It is not designed to prove a bucket is clean, and reading it that way is the root of most Macie misunderstandings.

Sensitive data discovery jobs are the depth tool. You choose the buckets, scope by prefix, extension, size or object tag, choose the identifiers, and run once or on a schedule. This is what you point at the bucket automated discovery just flagged.

# One-time job over a single prefix, recommended identifiers, 30% sampling
aws macie2 create-classification-job 
  --job-type ONE_TIME 
  --name "exports-prefix-review" 
  --managed-data-identifier-selector RECOMMENDED 
  --sampling-percentage 30 
  --s3-job-definition '{
    "bucketDefinitions": [
      {"accountId": "111122223333", "buckets": ["your-bucket-name"]}
    ],
    "scoping": {
      "includes": {
        "and": [
          {"simpleScopeTerm": {
             "comparator": "STARTS_WITH",
             "key": "OBJECT_KEY",
             "values": ["exports/"]
          }}
        ]
      }
    }
  }'

One detail about --sampling-percentage that trips people up: it selects a random percentage of eligible objects and then analyzes each selected object completely. It does not read a fraction of each file. For a bucket where every object comes from one pipeline, 20 or 30 percent tells you what you need. For heterogeneous uploads, sampling is a coin flip and you want the full pass.

What actually drives the bill

Macie bills on three dimensions, and the one that surprises people is not the one they budget for.

• Bucket evaluation. Per general purpose bucket monitored, prorated daily, up to the account ceiling. Predictable, driven by bucket count rather than data volume.
• Object monitoring. Per object tracked while automated discovery is enabled. This is inventory bookkeeping, not content reading, and it runs whether or not anything gets scanned that day. On an account with a log archive holding hundreds of millions of tiny objects, it can dwarf the other two.
• Data inspected. Per GB actually analyzed, covering both automated discovery and jobs.

So excluding buckets from automated discovery is a genuine cost lever, not just noise reduction. Log archives, CloudTrail destinations, build artifact stores and backup targets are usually safe exclusions. Exclude for the right reason though: a backup bucket holding database dumps of your customer table is exactly what you want scanned, however boring its name sounds.

The three dimensions appear as distinct usage types in the Cost and Usage Report, so they split cleanly. A FinOps platform like Vantage or CloudZero, or just a Cost Explorer view grouped by usage type, tells you within a day whether object monitoring or inspection is driving the number. That changes the fix entirely: object monitoring is solved by scoping, inspection by sampling and scheduling.

Getting findings somewhere a human sees them

Findings that only exist in the Macie console get read during onboarding week and never again. Two things are worth wiring up on day one.

Publish findings outward. Macie emits findings to EventBridge and integrates with Security Hub. EventBridge earns its keep because you can filter on severity and finding type and route only what matters, whether that’s a Slack channel, a ticket, or a Lambda that tags the bucket. If you already centralize alerts in Grafana Cloud or similar, that path keeps sensitive data findings next to everything else on call sees.

Configure the discovery results repository. Findings tell you what was found and where. The detailed discovery results, including analysis logs for objects where nothing was found, need an S3 bucket and a KMS key configured to persist. Nothing is retained long term until you set this up, and “we scanned it and found nothing” is exactly the record an auditor asks for.

One operational note on the reveal feature, which retrieves sample occurrences so a human can confirm a finding is real. It needs a customer managed KMS key and it’s genuinely useful for triage. It also means someone is now looking at live customer PII, which belongs in your access model and audit trail. Same goes if an engineer downloads an object to investigate locally: that copy is unmanaged PII on a laptop and needs a secure deletion step, whether that’s a wipe tool from a vendor like O&O Software or a documented and enforced process. Findings triage has a habit of creating the exposure it was meant to prevent.

Troubleshooting a bucket that reports nothing

When Amazon Macie PII detection reports nothing for a bucket you have doubts about, work down this in order. Each step rules out one gate, and stopping at the first hit saves you the rest.

1. Check the sensitivity label. Not yet analyzed at score 50 means nothing was analyzed. Do not read that as clean.
2. Check the coverage page. Macie names the issue directly: access denied, invalid encryption, invalid KMS key, permission denied, or unclassifiable.
3. Compare classifiable and unclassifiable counts with the describe-buckets query above.
4. List a few object keys. Run aws s3 ls against a prefix and look for extensions. Extensionless keys explain the whole thing.
5. Check the storage class. A lifecycle rule may have moved everything somewhere Macie doesn’t read.
6. Check encryption. SSE-C is unfixable without re-encryption. A customer managed KMS key is fixable with a key policy change.
7. Check object samples with list-resource-profile-artifacts to see exactly what was selected and skipped.
8. Only now question the identifiers. Run a targeted job with a custom identifier for the data you expect and see whether it fires.

Common mistakes

• Treating a low sensitivity score as evidence of no PII. It reflects what was found and how much was analyzed, so low coverage produces a low score.
• Enabling Macie in one region and assuming account-wide coverage. Settings and results are per region, every time.
• Running default identifiers in a business operating outside the jurisdictions the recommended set covers.
• Building custom identifiers from regex alone, then abandoning Macie over the false positive volume.
• Skipping the discovery results repository, then having no evidence trail when someone asks what was scanned and when.
• Excluding buckets by name pattern without checking contents. Backup and export buckets are frequently the highest-risk ones you own.

Best practices

• Measure coverage before findings. A coverage report is the first deliverable of a Macie rollout, not an afterthought.
• Fix extensions at the producer rather than building a rename pipeline downstream.
• Run the KMS permission analyzer script once per account, then again whenever a new customer managed key appears.
• Use automated discovery for breadth and targeted jobs for depth, letting the first choose targets for the second.
• Test every custom identifier against both a positive and a negative sample before it reaches a job.
• Curate allow lists early. The cost of a noisy first month is a team that stops reading findings.
• Re-check coverage after any change to bucket policies, KMS keys, lifecycle rules or ingestion pipelines. All four silently remove data from scope.

Frequently asked questions

Can Amazon Macie scan anything other than S3?

No. Macie analyzes objects in S3 general purpose buckets only. The documented workaround is bringing data to it: export RDS or Aurora snapshots to S3 in Parquet, or export a DynamoDB table to S3, then run a discovery job against the export. That works, but the export is now a second copy of your sensitive data with its own encryption and deletion requirements.

Does Macie scan new objects as they land?

Not per object. Automated discovery evaluates inventory on a daily cycle and prioritizes objects that are new or recently changed, so fresh data moves up the queue rather than triggering an immediate scan. If you need scanning tied to arrival, drive a discovery job from S3 event notifications through EventBridge and accept that you own that orchestration.

Why does a bucket I know has PII show a low sensitivity score?

Almost always coverage rather than detection. Check whether the objects are classifiable, whether Macie can decrypt them, and whether the label reads Not yet analyzed. If Macie genuinely read them and found nothing, the next suspect is the identifier set, and a targeted job with a custom identifier will tell you in one run.

Do I need both automated discovery and discovery jobs?

For most teams, yes. Automated discovery gives you the map at predictable cost. Jobs give you proof for a specific bucket at a specific time, which is what compliance evidence actually requires. Running only jobs means you never discover the bucket nobody told you about.

How do I cut false positives without missing real PII?

In order of preference: add keywords and a proximity rule so matches need context; use allow lists for specific known-benign values like your published contact details and test fixtures; and only then remove managed identifiers that are consistently wrong for your data. Removing identifiers is the bluntest option and the easiest to regret, so leave it last.

Does Macie work across accounts in an organization?

Yes, through AWS Organizations with a delegated administrator. Design around this: the administrator’s automated discovery settings apply to member accounts, members can’t change them, and members see coverage and sensitivity data for their own buckets but not the sensitive data findings themselves. That shapes who can actually remediate what.

The one thing worth remembering

Amazon Macie PII detection is only as good as its coverage, and coverage fails silently by design. An object with the wrong extension, the wrong storage class, or an unreachable encryption key produces no finding, and no finding renders identically to a clean result. Before you hand a Macie report to anyone who will make a decision from it, put the coverage numbers next to it: how many objects were classifiable, how many were analyzed, how many were skipped and why.

Do that once and the rest of the service becomes straightforward. Skip it and you’re maintaining an expensive dashboard reporting on whichever subset of your data happened to be readable.

Need help getting Macie to actually see your data?

Most of the work in a Macie rollout isn’t turning it on. It’s the unglamorous part: proving what got scanned, fixing the reasons things didn’t, and making the output land somewhere a human acts on it. That’s the part I help with.

• Coverage audits across your S3 estate, with per-bucket classifiable, analyzed and skipped counts and the specific cause of each gap
• Fixing those causes: KMS key policies, bucket policies blocking the service-linked role, lifecycle rules archiving data out of scope, pipelines writing extensionless objects
• Custom data identifiers and allow lists tuned against your real data and tested on positive and negative samples first
• Cost shaping: working out whether object monitoring or data inspection drives your bill, and scoping discovery so the number is defensible
• Findings pipelines through EventBridge or Security Hub into the alerting and ticketing you already run, with severity filtering that keeps signal above noise
• Multi-account setup under AWS Organizations, including the delegated administrator model and who can see and remediate what

If you want a second opinion, send me the output of the describe-buckets query above, or a screenshot of your coverage page, and I’ll tell you what’s actually being scanned.

The post Amazon Macie PII Detection: The Buckets It Never Opened appeared first on John Nessime.

That is the failure mode that actually bites when you build a medical lab data pipeline on AWS. Not the bucket you thought about. The seven places PHI ended up because a payload got logged, an error got queued, or a filename contained a medical record number.

This post walks through the design boundary by boundary: where lab data crosses from one trust zone into another, what leaks at each crossing, and which control closes it. It is aimed at engineers who have been handed an HL7 feed and a compliance checklist and told to make them meet. I am an engineer and not your compliance counsel, so treat the regulatory references here as pointers to read the primary text yourself.

Why laboratory data is a harder pipeline than it looks

Lab data has three properties that most ETL work does not.

First, results get corrected. A preliminary result goes out, the analyzer is recalibrated, and a corrected result follows hours or days later. Your pipeline is not append-only in the way a clickstream is. It has to carry amendment semantics all the way through, and an analytics table that silently keeps the first value is worse than no table at all.

Second, the wire format is old and the transport is older. Most laboratory information systems still emit HL7 version 2 messages, pipe-delimited segments carried over the Minimum Lower Layer Protocol, which is framed bytes over a raw TCP socket. AWS documents MLLP as the common transport standard for HL7 v2 interoperability in its Healthcare Industry Lens. It has no built-in authentication and no built-in encryption. Whatever security it has, you bolt on around it.

Third, the identifiers are load-bearing. Accession numbers, specimen IDs and medical record numbers are how the lab reconciles anything, so PHI is not a column you can drop. It is structural, and it will try to escape into every metadata surface you own.

Boundary one: getting messages out of the lab

The instrument or LIS sits on a network you probably do not control, run by a vendor who may not let you touch the configuration. That constrains the design more than anything on the AWS side.

You get three realistic shapes, and the LIS vendor usually picks for you.

• Live MLLP over a private link. An interface engine in the lab holds the TCP session and forwards over a Site-to-Site VPN or Direct Connect into a private subnet. Lowest latency, highest operational burden, because a dropped socket is a clinical incident and not a retry.
• Batch file drop over SFTP. The LIS writes flat files on a schedule and something picks them up. Far easier to reason about, far easier to audit, and acceptable whenever the downstream use is analytics rather than result delivery.
• An outbound API push. Newer systems will POST to an endpoint you expose. Best case, because you control authentication and you get a response code back, but rarest in practice.

Whichever shape you get, the same rule applies at this boundary: terminate the untrusted protocol as early as possible and convert to something you can authorize. A raw MLLP socket carries no identity. The moment those bytes land, wrap them in a message you can attribute, sign, and trace.

Two things go wrong here. First, people treat a network tunnel as authentication. A VPN tells you the packets came from the lab’s network. It does not tell you which system sent them, or that the sender was authorized to send that patient’s results. Second, people assume consumer VPN products fill the gap. Tools like NordVPN or Surfshark are fine for your own admin laptop on untrusted wifi, but they carry no business associate agreement and they are not a substitute for AWS Site-to-Site VPN or AWS Client VPN on a clinical path.

One piece of housekeeping that is cheap to do and expensive to skip: the business associate addendum. AWS makes this self-service through AWS Artifact, and it can be accepted across an entire organization. Only services on the current HIPAA Eligible Services Reference may hold PHI, and that list changes, so check it against the services you actually chose rather than your memory of it.

Boundary two: the landing zone

Raw messages land in S3. This part is well documented, so I will focus on the parts people configure once and never verify.

Make the bucket policy do the enforcing

Default bucket encryption means objects get encrypted. It does not mean a caller cannot write an object with different settings, and it does not stop a plaintext HTTP request. Two explicit denials cover the gap. The first protects the transport, the second protects the storage:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::lab-raw-inbound",
        "arn:aws:s3:::lab-raw-inbound/*"
      ],
      "Condition": {
        "Bool": { "aws:SecureTransport": "false" }
      }
    },
    {
      "Sid": "DenyWrongKey",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::lab-raw-inbound/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption-aws-kms-key-id":
            "arn:aws:kms:REGION:ACCOUNT:key/KEY-ID"
        }
      }
    }
  ]
}

Use a customer managed KMS key, not the AWS managed one. The reason is not stronger cryptography, it is the key policy. A customer managed key gives you a second, independent authorization surface: even a principal with broad S3 permissions cannot read the object if the key policy does not grant them decrypt. That separation is what makes a blast radius argument credible to an assessor, and it is what makes crypto shredding possible later.

Your object keys are metadata, and metadata leaks

This is the one I would put on a poster. Object contents are encrypted. Object keys are not. They show up in CloudTrail data events, S3 server access logs, S3 Inventory reports, bucket listings, Lambda event payloads, and every console screenshot anyone pastes into a ticket.

# Leaks PHI into six places that are not the bucket
raw/2024/ORU_MRN4471902_SMITH_JOHN_A1C.hl7

# Same object, no identifiers in the key path
raw/dt=<partition>/src=lis01/msg=01HXYZ...ULID.hl7

Use an opaque, sortable identifier and keep the mapping from identifier to patient inside an encrypted store you actually control. Partition on ingest date and source system, never on anything derived from the patient. This one decision removes more PHI surface than any other single change in the pipeline.

Immutability, and the trap inside it

Clinical laboratories in the United States operating under CLIA have record retention obligations set out in 42 CFR 493.1105. The floor for test reports is at least two years after the date of reporting, with pathology test reports at ten years, and several state requirements sit above the federal floor. Retention is therefore not something you invent; go and read the section, then check your state.

S3 Object Lock gives you write-once-read-many enforcement, and it is the right tool. But understand the two modes before you commit, because one of them is genuinely irreversible.

• Governance mode blocks deletion for most principals, but anyone holding s3:BypassGovernanceRetention can override it by sending the x-amz-bypass-governance-retention:true header. Note the S3 console includes that header by default.
• Compliance mode cannot be overridden by anyone, including the account root. Per AWS documentation, the only way to delete an object under compliance retention before its date expires is to delete the AWS account.

A typo in a retention date under compliance mode is permanent, and you pay to store the mistake for its full term. Object Lock also has to be enabled at bucket creation, alongside versioning. Start in governance mode, prove the retention values against real traffic, then move to compliance mode deliberately rather than as a default.

# Indefinite hold for a specific object version, e.g. under litigation
aws s3api put-object-legal-hold 
  --bucket lab-raw-inbound 
  --key raw/dt=.../msg=01HXYZ.hl7 
  --legal-hold Status=ON

Boundary three: where PHI escapes into your observability

Here is the section that justifies the post. Everything above is on the compliance checklist. This is the part that is invisible until an audit or an incident makes it visible.

Your operational plane is a PHI store you did not declare. Specifically:

• Application logs. One print(record) left in a parser during debugging writes patient demographics into CloudWatch Logs, where the default is often no expiry at all.
• Exception traces. Unhandled parse errors frequently include the offending segment in the message. That segment is the PHI.
• Step Functions execution history. If you pass message bodies between states as payloads, that content is retained in the execution record and readable by anyone who can view executions.
• Dead letter queues. The whole point of a DLQ is to keep the failed message. A DLQ full of unparseable HL7 is a PHI archive with a different access policy than the bucket you designed so carefully.
• Third-party monitoring. Anything shipped to an external platform leaves your boundary. Grafana Cloud, Datadog and equivalents are perfectly reasonable choices for infrastructure telemetry, but a log line containing PHI going to a vendor without an executed BAA is a disclosure, not a metric.
• Notifications. An SNS topic that emails an on-call engineer “failed to process record for J. Smith, DOB …” has just sent PHI to a mail server nobody reviewed.

The pattern that fixes it: pass pointers, not payloads

Adopt a single rule and enforce it in review: PHI moves by reference between components, never by value.

1. The ingest function writes the raw message to the encrypted bucket and emits only an opaque object key plus a message ID.
2. Every downstream state, queue and event carries that pointer. Nothing carries the record body.
3. Error handling logs the pointer and a failure class, never the content that failed.
4. A DLQ therefore contains pointers to failures, and reprocessing means re-reading from the bucket under the same key policy as everything else.

The cost of this pattern is a slightly noisier debugging experience. You cannot read the failing record straight out of the queue. You have to go and fetch it, with credentials, and that fetch is logged. That inconvenience is the control working.

Then close the log surface itself

Log groups created implicitly by Lambda and other services often retain data indefinitely unless you set a policy. Find the ones nobody configured:

# List log groups with no retention configured
aws logs describe-log-groups 
  --query 'logGroups[?retentionInDays==null].logGroupName' 
  --output text

# Set an explicit retention on one of them
aws logs put-retention-policy 
  --log-group-name /aws/lambda/hl7-ingest 
  --retention-in-days 30

Run the first command as a scheduled check, not a one-off. New functions create new log groups, and the default comes back every time someone deploys.

Amazon Macie is worth pointing at your buckets as a detective backstop. It will not stop a leak, but it will tell you when something started landing where it should not.

Boundary four: from identified data to analytics

At some point someone wants to query the data. This is the crossing where teams are most likely to convince themselves they have done something they have not.

Amazon Comprehend Medical will detect PHI entities in unstructured clinical text through its DetectPHI operation, returning entity types such as name, address, identifier and date, each with a confidence score. It is a genuinely useful tool for narrative fields: pathology comments, specimen notes, microbiology free text.

aws comprehendmedical detect-phi 
  --text "Specimen received from the referring clinic on the stated date."

Now the part the marketing pages underplay, straight from the AWS developer guide: the entities Comprehend Medical detects do not map one to one to the identifier list specified by the Safe Harbor method, and AWS explicitly recommends additional human review or other methods to confirm accuracy for compliance use cases.

Read that as: PHI detection is not de-identification. Under the HIPAA Privacy Rule, de-identification has two defined routes, Safe Harbor and Expert Determination. An NLP model with confidence scores is an input to either, not a substitute for either. If your analytics tier is meant to hold de-identified data, someone has to own that determination, and it will not be your pipeline.

What works structurally:

• Put the analytics tier in a separate AWS account with its own KMS key. Not a separate bucket. A separate account, so the blast radius argument is enforced by IAM boundaries rather than by convention.
• Make the flow one directional. The de-identification job reads from the identified side and writes to the analytics side. Nothing in the analytics account holds permissions pointing back.
• Replace identifiers with pseudonyms rather than deleting them, keeping the crosswalk in the identified account. This preserves your ability to join across results and to re-identify under a documented process, which the research use case usually needs.
• Watch quasi-identifiers. Rare test panels, unusual reference ranges and precise timestamps re-identify people in small populations even after the obvious fields are gone. This is a statistics problem, not an IAM problem.
• Use Lake Formation or equivalent for column and row filtering if analysts need partial access, so the restriction lives with the catalog rather than in each query tool.

Boundary five: sending results back out

Delivery is where clinical correctness and security pull against each other, and security usually loses quietly.

The correction problem is the sharp edge. HL7 result messages carry a status that distinguishes preliminary, final and corrected results. If your delivery layer is idempotent on message ID alone, a corrected result with a new ID for the same observation will happily land beside the original instead of superseding it. Two conflicting values, both marked delivered, nobody alerted.

Build the idempotency key from the identity of the observation, which typically means the placer or filler order identifier combined with the specific observation identifier, and carry the result status as a first-class field so a correction overwrites rather than appends. Keep every version, expose the current one.

For the outbound path itself:

• Presigned URLs are convenient for report PDFs and they are also a bearer credential. Anyone holding the link has the document. Keep expiry short and treat generation as an auditable event.
• If you expose a provider-facing portal, put a WAF in front of it. AWS WAF and Cloudflare are both credible here; the deciding factor is usually where the rest of your edge already lives, not the rule engines.
• Never email a result. Email a notification that a result is available, behind authentication. The distinction sounds pedantic right up until someone forwards a mailbox.

Boundary six: deletion, and the conflict nobody plans for

Two obligations point in opposite directions. Retention rules say keep the record. Privacy rights and internal policy say be able to remove data. If you set compliance-mode Object Lock across everything, you have chosen one side without noticing.

Resolve it by classifying before you lock. Records that fall under laboratory retention obligations go into the immutable tier with a retention period derived from the applicable rule. Derived artifacts, caches, intermediate parquet, analytics extracts and enrichment outputs do not belong there. They belong in buckets with lifecycle rules, and they are what you actually delete.

Crypto shredding covers the middle ground: if a data set is encrypted under a key used for nothing else, scheduling that key for deletion renders the ciphertext unrecoverable without touching the objects. It is clean, it is verifiable, and it only works if you planned the key granularity up front. Retrofitting per-tenant or per-cohort keys onto a pipeline that used one key for everything is a full re-encryption exercise.

Do not forget the endpoints. Instrument workstations, the interface engine box, and the laptop somebody used to test the parser all accumulate copies. Cloud controls do nothing for physical media, and secure erase tooling such as O&O SafeErase exists for exactly that step in a decommissioning runbook.

Troubleshooting a medical lab data pipeline on AWS

Messages arrive but nothing appears downstream

Check the DLQ first, then check whether the DLQ itself has a consumer. A surprising number of pipelines have a correctly configured dead letter queue that nobody monitors, so failures accumulate silently and the only symptom is a gap in the data. Alarm on DLQ depth greater than zero, not on a threshold.

Access denied from a Lambda that has the right IAM policy

With a customer managed KMS key, IAM permission on S3 is only half the grant. The function’s role also needs decrypt permission on the key, and the key policy has to allow it. If you are crossing accounts, both the key policy and the role policy must permit the action. This is the intended behavior of the two-lock design, and it is the single most common support question on any pipeline built this way.

Deletion fails on an object you are certain you should be able to delete

Check for a legal hold before you check retention. A legal hold has no expiry and is independent of the retention period, so an object can be past its retention date and still undeletable. Also confirm which mode the retention uses, because governance and compliance produce the same error to a caller without bypass permission.

Counts do not reconcile with the LIS

Compare on observations, not messages. One message can carry several results, corrections create additional messages for the same observation, and a naive message count will diverge from the LIS by exactly the amount that matters clinically. Reconcile daily and alert on drift rather than investigating at quarter end.

Timestamps drift between systems

HL7 v2 timestamps do not always carry a timezone offset, and analyzers are often set to local time with no daylight saving handling. Capture the offset at ingest from the source system configuration and store everything in UTC with the original string preserved. Recovering an ambiguous timestamp after the fact is unpleasant, and around a daylight saving transition it may not be possible at all.

Common mistakes

• Treating “HIPAA eligible” as “compliant.” Eligibility means AWS will cover the service under the BAA. Configuration is still entirely yours.
• Putting identifiers in object keys, queue names, or log group names. Encrypted contents, plaintext metadata.
• Enabling compliance-mode Object Lock before validating retention values. There is no undo, and no support ticket that fixes it.
• Assuming automated PHI detection equals de-identification. AWS itself says the entity list does not map one to one to Safe Harbor identifiers.
• Running the analytics tier in the same account as identified data. A bucket boundary is a policy away from collapsing. An account boundary is not.
• Shipping application logs to an external platform without checking what the log lines contain. Test it by grepping your own log group for a known test patient name.
• Ignoring corrected results until an analyst notices. Amendment handling is a day-one requirement, not a phase two feature.

Best practices worth the effort

• Write the data flow diagram first and mark every trust boundary crossing. Every arrow that crosses one needs a named control. This exercise finds more problems than any scanner.
• Encode the eligible service list as policy. Service control policies at the organization level stop someone from putting PHI into a service you never assessed.
• Keep VPC endpoints on the PHI path. Interface and gateway endpoints keep traffic to AWS services off the public internet and give you an endpoint policy as an extra choke point. Check which of your chosen services offer one.
• Build with synthetic HL7 from day one. Nobody should need production PHI to develop a parser. A cheap VPS from Contabo or InterServer is fine for a synthetic-data development box, and keeping that environment entirely outside the PHI boundary is the point.
• Alarm on absence. A feed that stops is more dangerous than a feed that errors, because errors are loud. Alert when expected message volume for a source drops below its floor.
• Test restores, not backups. Restore into an isolated account, confirm the KMS grants work there, and document how long it took.
• Review CloudTrail for data events on the PHI buckets. Object-level logging costs money and is the only record of who read what.

FAQ

Do I need a BAA with AWS before I start building?

Before PHI touches the environment, yes. You can architect and test with synthetic data first. AWS provides the business associate addendum through AWS Artifact as a self-service acceptance that can be applied across an organization, and only services listed on the HIPAA Eligible Services Reference may process, store or transmit PHI.

Should I use AWS HealthLake or build on S3, Glue and Athena?

HealthLake is a managed FHIR datastore with FHIR APIs, medical NLP and query built in, and it is the shorter path if your consumers speak FHIR. It does not natively ingest HL7 v2; AWS points to partner tooling or your own transformation for non-FHIR input. If your consumers are analysts with SQL and the data is structured lab results rather than clinical documents, an S3 and Athena lakehouse is simpler and easier to keep inside a boundary you control. Decide on the consumer, not the source.

How long do I have to keep laboratory records?

For CLIA-regulated laboratories in the United States, 42 CFR 493.1105 sets the federal floor, including at least two years for test reports after the date of reporting and at least ten years for pathology test reports. State law frequently exceeds these minimums, so the applicable period is whichever is longer for your jurisdiction. Read the regulation and confirm with your compliance lead before you encode a number into a retention policy.

Can Amazon Comprehend Medical de-identify data for me?

It can detect PHI entities and give you confidence scores, which is a strong starting point for redaction workflows. It does not by itself satisfy HIPAA’s de-identification standard. AWS documents that the detected entities do not map one to one to the Safe Harbor identifier list and recommends human review for compliance use cases. Treat it as a detector feeding a documented Safe Harbor or Expert Determination process.

Is S3 Object Lock compliance mode required for lab records?

Not inherently. The argument for compliance mode is a threat model in which a compromised or malicious administrator could remove retention, not a regulation that names the mode. Governance mode plus tightly controlled bypass permissions and strong audit logging is a defensible position for many laboratories. Choose deliberately and write down the reasoning, because compliance mode commits you to the storage cost for the full term with no exit.

Where does PHI most often leak in a pipeline that looks correctly configured?

Logs, dead letter queues, workflow execution histories, notification messages and object key names. All five are metadata surfaces that sit outside the storage layer everyone reviews. Grep your own log groups for a known test patient identifier and you will find out quickly whether your pipeline has the problem.

Do I need a dedicated AWS account just for the lab pipeline?

Separate accounts for identified and de-identified data are worth it even at small scale, because the boundary is then enforced by IAM rather than by naming conventions. Whether the PHI workload also needs isolating from your other production workloads depends on who holds admin access there. If the answer is “broadly the same people,” separate it.

The one thing to take away

A secure medical lab data pipeline on AWS is not defined by the encryption on its storage layer. Encryption is the easy part, and it is the part everyone gets right. What separates a design that survives an audit from one that does not is whether PHI can travel by value through your operational plane: into logs, into queue bodies, into workflow histories, into filenames, into alert emails.

Move PHI by reference, keep identifiers out of every metadata surface, and put an account boundary between identified and de-identified data. Do those three things and the rest of the compliance work becomes documentation rather than redesign.

Work with me on healthcare data pipelines

I design and review AWS data platforms for regulated workloads, and lab data is a specific enough problem that generic cloud advice tends to miss it. Things I can help with:

• Mapping every trust boundary in an existing pipeline and finding the PHI surfaces nobody documented, including logs, queues and object key schemes.
• Designing HL7 v2 or file-drop ingestion into S3 with KMS key separation, VPC endpoints and bucket policies that enforce rather than suggest.
• Building the identified and de-identified account split, including the one-way transformation job and the pseudonym crosswalk.
• Getting amendment and correction handling right so corrected results supersede rather than duplicate, all the way to the analytics tables.
• Retention and deletion design: Object Lock modes, lifecycle policies, crypto shredding key granularity, and the runbook that ties them together.
• Observability that is useful without being a disclosure risk, including CloudWatch retention hygiene and safe alerting patterns.

If you have an architecture diagram, a Terraform plan, or a redacted sample of the messages you are receiving, send it over and I will tell you what I would change first.

The post Where PHI Escapes: Building a Secure Medical Lab Data Pipeline on AWS appeared first on John Nessime.