It granted iam:PassRole on one role ARN and lambda:CreateFunction in the same document. That is not a wildcard and it does not read as administrative access. It is administrative access anyway, to whatever that role can do, for anyone who can write a Lambda function.

This is the shape of the problem with AI-generated IAM policies. The failure is almost never the obvious one. Models have absorbed enough style guidance to avoid "Action": "*", so what comes back passes the eyeball test and the linter and still hands over more than you meant to give. The dangerous output is the one that looks careful.

This post covers what policy generation genuinely does well, the failure shapes that survive human review, and where the hard boundary belongs: not in the review step, which is fallible, but in a mechanism that cannot be talked out of its answer.

What AI-generated IAM policies get right

Start with the honest case, because it is a strong one. IAM has thousands of actions across hundreds of services and the naming is inconsistent in ways nobody holds in their head. Some services use Describe, some use Get, some use both for different things. ARN formats vary per service and per resource type inside a service. Condition keys apply to some actions and not others.

A model is good at that recall problem, and at shape work: splitting a monolithic statement into per-resource statements, adding conditions, converting an inline policy into Terraform. Paste an AccessDenied message and you get a plausible read of which layer refused. So the argument is not that the tooling is useless. It is that recall and reasoning about consequences are different capabilities, and IAM is a system where the second one protects you.

The failure modes that survive review

Permissions that compose into something bigger

Every action is individually defensible. The escalation lives in the combination.

The canonical case is iam:PassRole beside any service that runs code: Lambda, EC2, ECS, Glue. Passing a role to a compute service means your code executes with that role’s permissions, so your ceiling is not your own policy, it is whatever the most powerful passable role can do. Rhino Security Labs catalogued a long list of these chains and every one of them is built from permissions that look unremarkable alone.

Another that catches people: iam:CreatePolicyVersion on a policy you are attached to. A new version normally needs to be set as default to take effect, but the create call accepts a flag that makes it default immediately, and that flag does not require iam:SetDefaultPolicyVersion. A permission that reads as “can update policy documents” reads to an attacker as “can write myself an admin policy.”

Ask a model whether a policy grants admin and it will check the statements and say no, correctly, one at a time. Composition is the part it does not reliably do, and it is the part that matters.

Actions that do not exist

Plausible action names that no service publishes. s3:ListBucketContents instead of s3:ListBucket. IAM accepts a policy containing an unrecognized action string because the document is syntactically valid, so nothing fails at attach time.

What happens instead is worse. The application throws AccessDenied in production, somebody debugging under pressure decides the granular list must be wrong, and the fix is a service wildcard. The invented action does not grant too much. It causes a human to grant too much a week later, and by then nobody connects the two events.

Resource ARNs that look constrained and are not

An ARN with real characters in it feels safer than a star. Often it is not. The common versions are a trailing wildcard placed one segment too high, and an action whose resource element the service ignores. s3:ListAllMyBuckets is account-scoped no matter what you put in Resource. Several ec2:Describe actions behave the same way. A policy can read as resource-scoped throughout and be account-wide for the actions that count.

The related trap is the missing condition: cross-account trust policies without an external ID, bucket policies without aws:PrincipalOrgID, integration roles without aws:SourceIp where the caller has a stable egress address. Generated policies include conditions when you ask and omit them when you do not, because an absent constraint never produces an error.

Four things the model cannot know

Some of this is not a capability gap you can prompt past. It is missing input:

• Which resources exist. A bucket name is a guess unless you supplied it. Guessed names either fail closed or, if the wildcard is loose enough, match something you never intended.
• Which resources are sensitive. Your backup bucket and your public assets bucket are indistinguishable strings. Sensitivity lives nowhere in the document.
• What else is attached. Effective permissions come from identity policies, resource policies, boundaries, SCPs and session policies together. Reviewing one document tells you very little.
• What the blast radius is. “Can delete objects” means one thing for a scratch bucket and another for the only copy of a client’s data.

The training material is also skewed. Public IAM examples, including plenty of vendor documentation and countless forum answers, are permissive because permissive examples work on the first try. A model that has read the internet has read a lot of over-broad policies presented as correct.

Where the hard boundary belongs

Review is not a boundary. Review is a filter with a pass rate that drops the more policies you look at in a week, and generated policies are cheap to produce, so volume rises exactly when attention falls.

The boundary has to be something that says no without reading the document. AWS gives you two mechanisms and they are not interchangeable.

A permissions boundary is a managed policy attached to a user or role that caps what identity-based policies can grant it. It grants nothing itself. Effective permissions become the intersection of the identity policy and the boundary, so an identity policy allowing iam:* against a boundary that omits IAM produces no IAM access at all. If no boundary is attached, none is evaluated, so this is opt-in per identity.

A service control policy sets the maximum available permissions for every principal in an account or organizational unit, and also grants nothing. It is the right home for things nobody in that account should ever do, and it applies whether or not somebody remembered to attach a boundary.

The division I use: SCPs carry account-wide invariants, such as denying escalation-relevant IAM write actions to everyone outside the break-glass role and denying regions you do not operate in. Boundaries carry the per-workload ceiling, and they are what makes it safe to let a pipeline role create other roles. You can permit that role to create roles while requiring, through a condition on the boundary key, that every role it creates carries your boundary. The pipeline can then generate whatever policy it likes, because the ceiling is enforced at evaluation time rather than at review time.

If the only thing between a generated policy and production is a person reading JSON, you do not have a boundary. You have a habit.

The gate I would actually run

Mechanical checks in a pipeline, failing the build, not a checklist somebody is supposed to remember.

1. Lint for nonsense. Parliament catches unknown action names, resource formats that cannot match the action, and type mismatches. Cheapest way to catch an invented action before it becomes a wildcard.
2. Run AWS policy validation. Access Analyzer knows service-specific rules a generic linter does not.
3. Assert the escalation actions are absent. Name the actions you never want granted and fail if any appear. This is the check that maps to the composition failure.
4. Compare against the current policy. For edits, check whether the proposed document grants access the existing one does not. This one is billed per call, so run it on changed policies rather than the whole repository.
5. Scan for known risky patterns. Cloudsplaining scores policies against categories including privilege escalation, data exfiltration and resource exposure, and produces something you can hand to a client.

# 1. Lint. Exit status is non-zero when there are findings.
parliament --file policy.json

# 2. AWS-side validation
aws accessanalyzer validate-policy 
  --policy-document file://policy.json 
  --policy-type IDENTITY_POLICY

# 3. Fail if any escalation-relevant action is granted
aws accessanalyzer check-access-not-granted 
  --policy-document file://policy.json 
  --access actions="iam:PassRole","iam:CreatePolicyVersion","iam:AttachRolePolicy" 
  --policy-type IDENTITY_POLICY

# 4. Does the proposed policy grant more than the current one?
aws accessanalyzer check-no-new-access 
  --existing-policy-document file://current.json 
  --new-policy-document file://proposed.json 
  --policy-type IDENTITY_POLICY

# 5. Risk-categorised report
cloudsplaining scan-policy-file --input-file policy.json

Step three is the one to add first if you only add one. It turns “we reviewed it” into “the build fails.”

Attaching the boundary is a single call, and it is the part people skip because the role already works without it:

aws iam put-role-permissions-boundary 
  --role-name app-deploy-role 
  --permissions-boundary arn:aws:iam::123456789012:policy/WorkloadCeiling

Generate from observed behavior instead

There is a better source of truth than a model’s guess about what an application needs, and it is the application. Access Analyzer policy generation reads CloudTrail events for a role over a chosen window and writes a policy from what was actually called. Know its limits before relying on it:

• It analyzes up to ninety days of history and needs a trail already logging for the account, so a role that only exercises certain paths quarterly produces an incomplete policy.
• Coverage is per-service. For some services it identifies individual actions; for others it can only tell you the service was used and prompts you to fill in the actions yourself.
• It does not produce action-level detail for data events such as S3 object-level operations.
• iam:PassRole is not included in generated policies. If your workload needs it, a human adds it back by hand, so the most escalation-relevant line in the document is the one the generator did not write.

Behavior-derived policies also fail in a useful direction. A missing permission produces a denial you can see in CloudTrail, which beats a permission you did not know you granted.

Arguments that do not survive contact

• “We review every policy before merge.” Review quality is a function of volume and attention, and generation raises volume. Composition failures are also the hardest to spot by reading, because spotting them means knowing what other roles exist and what they can do.
• “Access Analyzer came back clean.” Validation checks the document. It does not know that the role ARN you may pass happens to have AdministratorAccess attached. Clean findings on a policy that grants a dangerous chain is the normal case, not an anomaly.
• “A permissions boundary is overkill for a small project.” Small projects are where this bites hardest. One account, one deploy role, no organization, nothing between a bad policy and everything you own. The boundary is one managed policy and one CLI call.
• “We will tighten it after launch.” Nobody does. If you genuinely intend to, drive it from Access Analyzer’s unused access findings so you have a work queue instead of an intention.
• “Just prompt the model to follow least privilege.” That changes the output surface, not the reasoning. Fewer wildcards, better statements, no evaluation of what the combination permits.

How I would decide

My rule is about which actions are in the document, not about who or what wrote it.

Ship generated policies freely when the document touches only data-plane actions on named resources, the identity has a boundary, and the pipeline checks pass. Read a bucket prefix, publish to a topic, write to a table. That is most policies, and hand-writing them wastes your afternoon.

Treat generated output as a first draft when the policy includes any IAM write action, any PassRole, any trust policy edit, any resource policy readable from outside the account, or any KMS key policy. Use it to save typing, then verify every statement against the service authorization reference yourself.

Do not generate at all for the boundary policies and SCPs. Those are the mechanism. Something that constrains everything else should be short, hand-written, understood by whoever owns the account, and changed rarely.

Two notes for small engagements. If you run application infrastructure on a VPS from a provider like Contabo or InterServer and pull AWS in for storage or mail, that server holds long-lived credentials somewhere your AWS controls do not reach, so scope its policy to a bucket prefix and attach a boundary before worrying about anything else. And if the credentials go to a SaaS connector, a marketing platform such as GoHighLevel or a payments integration, you are handing keys to a system whose behavior you cannot audit. Those are the policies to write by hand.

Frequently asked questions

Are AI-generated IAM policies safe to use in production?

For data-plane permissions on named resources, generally yes, provided the identity carries a permissions boundary and the policy passes automated validation. For anything involving IAM write actions, PassRole, trust policies or key policies, treat the output as a draft and verify each statement yourself. What matters is the blast radius of the actions, not the authorship.

Why does a generated policy pass validation and still grant too much?

Validation examines one document in isolation. Privilege escalation usually comes from a combination of individually reasonable permissions, or from the interaction between the policy and something outside it, such as which roles are passable and what those roles can do. A document can be internally correct and still open a path.

What is the difference between a permissions boundary and an SCP?

Both cap permissions and neither grants any. A boundary attaches to a single IAM user or role and limits what that identity’s policies can grant it. An SCP applies to every principal in an account or organizational unit and needs AWS Organizations. With one account and no organization, boundaries are what you have. With an organization, use both.

Can IAM Access Analyzer replace writing policies by hand?

Partly. Generation from CloudTrail activity beats a guess because it reflects what the workload actually called. It is limited by the analysis window, by per-service coverage differences, by the absence of action-level detail for data events, and by iam:PassRole being excluded. Treat it as a strong draft that needs a human pass over anything the generator could not see.

Which IAM actions should I block outright in a small account?

The ones that let a principal rewrite its own permissions or borrow another identity’s: creating and setting default policy versions, attaching and putting inline policies on users, roles and groups, creating access keys for other users, and passing roles to compute services. Deny them for everything except a break-glass identity, then grant them back deliberately where a workload genuinely needs one.

How do I catch an invented action name before production?

Run a linter that carries the IAM action catalogue. Parliament flags unknown actions and exits non-zero when it finds anything, so it drops straight into a pipeline step. Catching it early matters less because the fake action is dangerous and more because the eventual fix for the denial tends to be a service-level wildcard.

The one thing worth remembering

AI-generated IAM policies are not more dangerous because a model wrote them. They are more dangerous because they arrive faster than you can think about them, and because they arrive looking finished.

So stop trying to make the review step better. Put the boundary somewhere that does not depend on anyone paying attention: a permissions boundary on every identity a pipeline can create or modify, an SCP denying the escalation actions to everyone who does not need them, and a build step that fails when a forbidden action shows up. Then let generation happen at whatever speed it wants.

The policy approved in ninety seconds is not the failure. The absence of anything underneath it was.

Need help drawing that boundary in your account?

Most of the IAM work I take on is exactly this: an account that grew organically, permissions nobody wants to touch in case something breaks, and no ceiling underneath any of it. Things I can help with:

• Auditing existing roles and policies for privilege escalation chains, including the passable-role paths that never show up in a single-document review
• Designing and attaching permissions boundaries for pipeline, workload and human identities, including the delegation pattern that lets a deploy role create roles safely
• Writing the SCP set for a small organization: escalation-relevant IAM actions, region restrictions, and protection for logging and billing resources
• Adding Access Analyzer validation and custom policy checks to a CI pipeline so forbidden actions fail the build instead of relying on review
• Replacing over-broad roles with policies generated from CloudTrail activity, then closing the gaps the generator cannot see
• Scoping down credentials handed to third-party SaaS integrations and to servers outside AWS, where your account controls do not apply

If you want a second opinion on something specific, send me the policy JSON, the output of a failing check-access-not-granted run, or the role list from your account, and I will tell you what I would change and why.

The post AI-Generated IAM Policies: Where the Hard Boundary Belongs appeared first on John Nessime.

Moving the files is the solved part. Which bucket they land in, and what sits in front of that bucket, decides whether you spend almost nothing or spend more than the server you were trying to shrink.

Cloudflare R2 vs Amazon S3 usually gets reduced to one sentence: R2 does not charge for data transfer out, S3 does, therefore R2. That is wrong in both directions, and understanding why is most of the value here. This post covers the failure mode that shows up after the migration rather than during it, an honest profile of each service, the levers that actually move the number, and a procedure you can run against your own traffic in an afternoon.

Free egress is not free serving

R2 charges nothing to move bytes out, on any storage class. It still charges for the read. Every GetObject and every HeadObject is a Class B operation, and a media library made of thousands of small files racks those up fast. Cloudflare’s own asset hosting example in the R2 pricing docs makes the point: storage falls inside the free tier, writes fall inside the free tier, and the whole bill is read operations.

So the meter is not gigabytes. The meter is requests that reach the bucket, which means the variable deciding your bill on either platform is cache hit ratio.

This is where offloading setups go wrong without anyone noticing. The plugin rewrites image URLs to point at the bucket, images render, pages look fine, and every request for every thumbnail travels all the way to the origin and gets billed. Nothing breaks. Nothing alerts. The number just grows with traffic.

There is a sharper version on the R2 side. A new bucket gives you a public URL on an r2.dev subdomain, which is convenient and tempting. Cloudflare documents it as a testing endpoint with a variable rate limit. Push production traffic through it and requests get throttled with 429 Too Many Requests, throughput can be throttled too, and you get no cache, no WAF, no bot management. Those only exist once the bucket sits behind a custom domain you control. That is the invisible failure: not a broken image, a working setup billed on every request and rate limited under load.

Cloudflare R2: where it wins and where it hurts

Where R2 wins

• Egress really is zero. Not zero within a partner network, not zero up to a ratio of stored data. Zero, on both storage classes, through the S3 API and through Workers.
• A custom domain puts it behind the CDN. One managed CNAME and reads start being absorbed at the edge instead of hitting the bucket. No second product to buy and wire up.
• Deletes are free operations, along with aborting a multipart upload. Cleaning up a messy library costs nothing.
• Unauthorized requests are not billed. A caller without permission gets a 401 and you are not charged, which matters once someone starts hammering your bucket path.
• Migration tooling is free to use. Super Slurper copies a bucket across in bulk, Sippy migrates lazily on first miss. You pay only for the operations they perform against R2.

Where R2 hurts

• S3 compatible is not S3 identical. Cloudflare publishes a table of which operations are implemented and which are not. Real tooling has broken on the gaps: when AWS shipped SDKs that enabled CRC32 checksums by default, R2 rejected the header until the mismatch was resolved.
• There is no cold archive tier. Infrequent Access is as cold as it gets, and it carries a minimum storage duration, a retrieval fee, and doubled operation rates. For an active library that is the wrong tier, not a saving.
• Data lives in one primary location. Location hints are best effort rather than a region guarantee, and are honored only the first time a bucket with that name is created. Read performance away from that location comes from the edge cache, not from replicas.
• Plugin support is thinner. WP Offload Media, the long-standing WordPress option, officially lists Amazon S3, DigitalOcean Spaces, and Google Cloud Storage. R2 users generally land on Media Cloud, Advanced Media Offloader, Next3 Offload, or the S3-Uploads route driven from WP-CLI.

Amazon S3: where it wins and where it hurts

Where S3 wins

• Origin transfer to CloudFront is not billed. This quietly demolishes the usual comparison. AWS waives data transfer from an AWS origin to CloudFront, so a properly built S3 media stack never pays the direct-to-internet rate on cached traffic.
• The CloudFront free allowance is permanent, covering a monthly volume of data transfer out and requests rather than expiring after twelve months. Small sites can sit inside it indefinitely.
• Event driven derivatives are mature. Upload fires a Lambda that generates thumbnails, strips metadata, converts formats, writes the result back. Hard to beat if your media pipeline does real work.
• A full storage ladder with real lifecycle transitions, down to deep archive. If a large chunk of your library is genuinely cold, S3 has somewhere cheap to park it.
• Governance depth. Object Lock, versioning, replication, IAM condition keys, CloudTrail data events. When an auditor asks for a control, S3 usually has a named feature for it.

Where S3 hurts

• The default configuration is the expensive one. Point image URLs at the bucket endpoint and every byte leaves at the internet rate with nothing cached in front of it. The cheap path has to be built deliberately.
• Four meters instead of two: storage, bucket requests, CDN requests, CDN data transfer, plus per-feature charges. Forecasting takes real effort.
• Geography changes the rate. Delivery costs more per gigabyte to some regions than others, so an audience shift moves your bill without you changing anything.
• Leaving costs money. Copying a large library out is billed transfer, once, in a lump. Usually small against ongoing savings, but it is a real number somebody has to approve.

How to decide between Cloudflare R2 vs Amazon S3

Four levers move the outcome. Everything else is noise.

Cache hit ratio

The dominant term on both platforms, and the one people skip. At a high hit ratio the origin barely gets touched and both bills collapse toward the storage line. At a low hit ratio, R2 bills read operations and S3 bills origin GETs, and both climb with traffic. Media should be trivially cacheable because the files never change. If yours is not caching, that is a configuration problem worth fixing before you pick a vendor, because fixing it changes the answer.

Object count, not library size

WordPress does not store one file per upload. It stores the original plus every registered image size, and themes and page builders happily register more. Ten thousand uploads can easily be sixty thousand objects, each one a write on migration and a read on a cache miss. Count them before you model anything:

cd /path/to/wordpress

# Objects you are about to create in the bucket
find wp-content/uploads -type f | wc -l

# Bytes
du -sh wp-content/uploads

# How much of that count is derivative sizes rather than originals
find wp-content/uploads -type f -regextype posix-extended 
  -regex '.*-[0-9]+x[0-9]+.(jpe?g|png|webp|avif)$' | wc -l

That last number is usually the surprise. If most of your object count is thumbnails, trimming unused registered sizes before migrating is the cheapest optimization available, and it shrinks writes, reads, and storage at once.

How cold the library actually is

Most libraries are a long tail. A small set of recent files takes nearly all the traffic while the rest sits untouched for years. If that tail is large, S3 has somewhere genuinely cheap to put it and R2 does not. Be careful with R2’s Infrequent Access class here: it lowers the storage rate but raises both operation rates, adds a retrieval fee, and enforces a minimum duration whether you keep the object or not. It fits write-once, read-almost-never data, not a library anything still links to.

What the rest of your stack already is

If your application already runs in AWS and generates derivatives with Lambda on upload, moving to R2 means rebuilding that pipeline against Workers or R2 event notifications. That is real work you do not get paid for. If your site is a VPS from somewhere like InterServer or Contabo with Cloudflare already in front of it, R2 is nearly free work: the DNS is there, the cache is there, and the bucket slots in behind a subdomain you already control.

Setup details that change the answer

A few configuration choices matter more than the vendor choice. Get these wrong and the cheaper platform produces the bigger bill.

Serve from a custom domain, always

On R2 this is the difference between a cached, protected asset host and a throttled test endpoint. Attach one from the dashboard or from Wrangler:

npx wrangler r2 bucket domain add my-media-bucket 
  --domain=cdn.example.com 
  --zone-id=<YOUR_ZONE_ID>

npx wrangler r2 bucket domain list my-media-bucket

Then disable public access on the r2.dev subdomain, or you have left a second uncached, unprotected door into the same objects.

Verify the cache instead of assuming it

Assumptions about caching are where the money leaks. Ask the edge directly: request the same object twice and read the headers.

# Cloudflare in front of R2
curl -sI https://cdn.example.com/2024/07/photo-1024x768.jpg 
  | grep -iE 'cf-cache-status|cache-control|age'

# CloudFront in front of S3
curl -sI https://cdn.example.com/2024/07/photo-1024x768.jpg 
  | grep -iE 'x-cache|cache-control|age'

First request, expect a miss. Second, expect a hit. If the second still reports a miss, or reports a status meaning the response was never eligible for caching, every image view is costing you an origin read. Fix that before you compare anything.

Long cache lifetimes and versioned filenames

Uploaded media is immutable in practice. Nobody edits the bytes of photo-1024x768.jpg, they upload a new file. So cache lifetimes should be long and the object name should change when the content does. If you find yourself purging image caches regularly, the real problem is that your filenames are not versioned.

On the Cloudflare side, Smart Tiered Cache is worth enabling for R2 origins. It routes edge misses through an upper tier data center close to your bucket instead of letting every edge location fetch independently, which cuts the number of requests reaching R2 at all.

Clean up failed uploads

Large uploads use multipart. Interrupted ones leave orphaned parts that occupy billed storage and do not show up in a normal object listing. Both platforms support lifecycle rules for this, and on R2 you can manage them from Wrangler:

npx wrangler r2 bucket lifecycle list my-media-bucket

npx wrangler r2 bucket lifecycle add my-media-bucket 
  --name=expire-temp 
  --prefix=tmp/ 
  --expire-days=30

Configure an abort rule for incomplete multipart uploads on day one, whichever platform you land on. It is the most common source of storage you pay for and cannot see.

A decision procedure you can run this afternoon

1. Measure your current cache hit ratio for image paths, from analytics or a sampled read of access logs. Everything downstream depends on this number.
2. Count objects, not gigabytes. Run the find commands above and separate originals from derivative sizes.
3. Estimate monthly origin reads as total image requests multiplied by the miss rate. That, not your bandwidth, is what you feed into either pricing calculator.
4. Treat writes as a one-off migration spike plus a modest steady rate. Migration is usually the largest write event the bucket ever sees.
5. Price both against current published rates on the same day, using Cloudflare’s R2 calculator and the AWS pricing calculator. Rates move. Do not trust a number copied from a blog post, including this one.
6. Test with a prefix before committing. Offload one year of uploads, point the site at it, watch cache status headers and the operations dashboard for a week. A week of real traffic beats any spreadsheet.

If it comes out close, take R2. Not because it is cheaper in that scenario, but because it has fewer meters to reason about, and a simpler operational model is worth something on the day something breaks.

Arguments that don’t survive contact

“S3 charges egress, R2 doesn’t, so R2 always wins”

Only true against the worst possible S3 setup. Transfer from an S3 bucket to CloudFront is not billed, so a properly built S3 media stack pays CDN delivery rates on cache misses, not the direct-from-bucket rate. Add CloudFront’s permanent free allowance and a small site pays nothing on either platform. The gap opens at scale and on origin-heavy traffic, not on the first terabyte.

“R2 is a CDN”

R2 is object storage that can sit behind a CDN you already have. The caching, the WAF, the bot rules all come from the Cloudflare zone, and none of it applies to the development URL. If the bucket is not behind a custom domain on a zone you control, you have storage without delivery.

“S3 compatible means drop-in”

It means most tools work with a changed endpoint and a region value of auto. It does not mean every operation, header, and checksum behaves identically. Budget an afternoon for fighting an SDK default, and test uploads, multipart uploads, deletes, and signed URLs against the real bucket before cutting over.

“We’ll save money moving cold files to Infrequent Access”

Sometimes. The lower storage rate arrives with higher operation rates, a retrieval fee, and a minimum duration. If files are cold enough to justify that, they are probably cold enough to belong in a real archive tier, which R2 does not have. Model it against your read pattern rather than assuming a cheaper per-gigabyte number is a cheaper bill.

Frequently asked questions

Is Cloudflare R2 cheaper than S3 for a WordPress media library?

Usually, once traffic is meaningful and the library is served publicly. For a small site behind a well configured CDN, both can land near zero. R2’s advantage grows with egress volume and with the number of requests that miss cache, which is why measuring hit ratio comes first.

Can I use the same WordPress plugin for both?

Several plugins speak the S3 API and support both, including Media Cloud, Advanced Media Offloader, and Next3 Offload. WP Offload Media officially lists Amazon S3, DigitalOcean Spaces, and Google Cloud Storage, so check current provider support before assuming a swap is free. Whichever you pick, confirm it rewrites existing URLs and not only new uploads.

How do I migrate an existing S3 bucket to R2 without downtime?

Cloudflare offers two paths. Super Slurper copies everything in bulk. Sippy migrates incrementally, pulling an object from the source the first time it is requested and serving from R2 afterward. Both are free to use, you pay for the operations they perform against R2, and your source bucket may charge you for the reads.

Do I still need a CDN in front of R2?

You need a custom domain on a Cloudflare zone, which is what puts the CDN in front of it. That is not optional for production. The development URL is rate limited by design and gets no cache, WAF, or bot management.

Can I keep S3 and just put Cloudflare in front of it?

You can, and it does cut origin reads. Be aware that cache misses then pull from S3 across the public internet, billed as ordinary S3 egress, unlike an S3 origin sitting behind CloudFront. Reasonable as an interim step, rarely the cheapest end state.

What about Backblaze B2, Wasabi, or DigitalOcean Spaces?

All viable and worth pricing if you are already modelling. B2 competes on raw storage, Wasabi sells predictability, Spaces is convenient when your droplets are already there. The analysis transfers directly: find the meters, find your cache hit ratio, multiply.

How do I monitor the bill after migrating?

Watch operations and cache hit ratio, not storage. Storage grows slowly and predictably. Operations track traffic and configuration mistakes, which is where the unpleasant surprises live. A Grafana Cloud dashboard or a scheduled pull from the provider’s usage API is enough; the point is that somebody looks weekly rather than at invoice time.

Conclusion

If you take one thing from this Cloudflare R2 vs Amazon S3 comparison, take this: the vendor choice is second order. What decides your media offloading bill is how many requests reach the bucket, and that is a function of your cache configuration, your object count, and whether you put a real domain in front of the storage.

Fix the cache first. Count the objects. Then price both, on the same day, against your own numbers. Most of the time R2 comes out ahead for public media, S3 comes out ahead when the library is entangled with AWS services or needs a genuine archive tier, and the difference is smaller than the internet suggests.

Need help moving a media library without breaking it?

Media offloading looks like a plugin install and turns into a URL rewriting, caching, and permissions problem. I work with teams on the parts that are easy to get subtly wrong:

• Modelling R2 against S3 using your real object counts, request volumes, and cache hit ratio instead of a generic calculator
• Planning and running the migration: bulk or incremental copy, URL rewriting, and a rollback path if the cutover misbehaves
• Custom domain, cache rule, and header configuration so origin reads collapse instead of tracking your traffic
• Bucket permissions, CORS, signed URL flows for private downloads, and lifecycle rules for orphaned multipart uploads
• Trimming registered image sizes and derivative sprawl before migration so you stop paying to store thumbnails nothing links to
• Dashboards for operations, cache hit ratio, and storage growth so the bill stops being a monthly surprise

Send me a curl -I of one of your image URLs and a rough object count, and I can usually tell you quickly whether you have a pricing problem or a caching problem.

The post Cloudflare R2 vs Amazon S3 for Media Offloading: Where the Bill Actually Comes From appeared first on John Nessime.

That gap is the entire problem. Run status tells you the code executed. It tells you nothing about whether data arrived. Data pipeline freshness monitoring is the signal that closes the gap, and most teams add it only after an analyst finds the stale table first.

This post covers the three signals worth instrumenting for a batch or streaming pipeline: how current the data is, how long it took to get there, and how often the thing breaks. For each one I’ll cover what to measure, where to measure it, the queries and config that produce it, and the specific ways the measurement lies to you.

The signal that lies to you first: run status

Job success is a statement about your code path, not about your data. A pipeline that reads an empty S3 prefix, transforms zero rows, writes zero rows and exits cleanly has succeeded. So has one whose upstream API silently started returning an empty page after a token rotation. So has one whose incremental watermark got stuck and now re-reads the same already-loaded slice on every run.

All three are green. All three are producing stale data. This is why the ordering matters: freshness is the signal that fires first in a real incident, and run status is usually the last one to notice anything at all.

So keep run status. It’s cheap and it catches crashes. Just stop treating it as your top-line health indicator.

Freshness: measure the table, not the job

Freshness is the age of the newest record in a dataset. It’s measured against the dataset itself, not against the process that filled it, and that distinction is what makes it useful. If your job disappears entirely, freshness keeps climbing and keeps alerting. If your job succeeds while doing nothing, freshness keeps climbing and keeps alerting. It’s the one signal that survives both failure modes.

Two clocks, and you need both

Every row usually carries two timestamps: when the event happened in the source system, and when your pipeline wrote it. Track the maximum of each.

• Max event time tells you whether the upstream system is still producing. If this stops moving, the problem is upstream of you.
• Max load time tells you whether your pipeline is still writing. If this stops moving while event time is fine, the problem is yours.

Watching only one of them means every stale-data page starts with twenty minutes of figuring out which side of the boundary the fault sits on. Watching both answers that in the alert body.

A freshness probe is a small scheduled query. This one is PostgreSQL syntax, and it returns both clocks plus a volume check in a single round trip:

select
  'orders' as dataset,
  extract(epoch from max(event_ts))  as max_event_ts,
  extract(epoch from max(loaded_at)) as max_loaded_at,
  count(*) filter (where loaded_at >= now() - interval '1 hour') as rows_last_hour
from analytics.orders;

The row count matters. A pipeline can advance its load timestamp while writing almost nothing, which is what a partially broken source looks like. Freshness alone will not catch that; freshness plus volume will.

Run that probe on a schedule that’s independent of the pipeline, push the results as gauges, and alert on age in PromQL:

# Newest row is older than 90 minutes
time() - max by (dataset) (dataset_max_loaded_timestamp_seconds) > 5400

# The probe itself has stopped reporting: a deadman check
absent(dataset_max_loaded_timestamp_seconds{dataset="orders"})

That second rule is the one people forget. A freshness metric that disappears looks identical to a healthy silence on a graph. absent() is what turns “no data” into a page.

Where dbt fits

If you’re already running dbt, source freshness is built in and worth using before you write anything custom. You declare thresholds per source, and dbt queries the maximum of your timestamp column and compares it to now.

sources:
  - name: raw_shop
    schema: raw
    config:
      loaded_at_field: _loaded_at
      freshness:
        warn_after: {count: 2, period: hour}
        error_after: {count: 6, period: hour}
    tables:
      - name: orders
      - name: refunds
        config:
          freshness:
            warn_after: {count: 12, period: hour}
            error_after: {count: 24, period: hour}

Three details that trip people up. First, dbt source freshness is a separate command; dbt build does not run it, so a green build says nothing about source staleness. Second, it exits non-zero when a source hits its error_after threshold, which makes it a natural gate at the top of a job: fail fast rather than building models on stale input. Third, results land in target/sources.json, which is the artifact you parse if you want to distinguish warn from error, or ship the numbers into Prometheus rather than just failing the run.

dbt source freshness --select source:raw_shop --output target/freshness.json

One caveat on the YAML above: dbt moved these keys under a config: block in recent releases, and loaded_at_field followed later. Older projects nest them directly under the source. Check what your project’s version expects before copying, because a misplaced key fails quietly by simply not calculating freshness at all.

Latency: name the clock before you name the number

“Our pipeline latency is twelve minutes” is meaningless until you say which two points you measured between. There are at least three plausible definitions, and teams routinely argue past each other because they’re each using a different one.

• Run duration. Start to end of the job. Easy, and mostly useless for anyone downstream.
• Ingestion latency. Source extract to target load. This is what you control.
• End-to-end latency. Event time to the moment the row is queryable. This is what the business actually feels, and it includes queue wait, scheduler delay, and every upstream hop you don’t own.

Publish end-to-end as the headline and keep the others as breakdown. If you only track run duration, a scheduler backlog that delays every run by forty minutes is completely invisible to you: each individual run still takes eleven minutes.

Instrument per stage with a histogram so you can ask percentile questions later without re-instrumenting:

histogram_quantile(
  0.95,
  sum by (le, stage) (rate(pipeline_stage_duration_seconds_bucket[6h]))
)

Use the median for capacity planning and p95 or p99 for the SLO. Averages hide the exact tail that generates the complaints, and on a pipeline that runs a few dozen times a day, a single pathological run is a real fraction of your day.

For streaming, the equivalent is consumer lag expressed in time rather than offsets. Offset lag of fifty thousand messages means nothing without a rate; two hundred seconds of lag means something to everyone. Kafka Lag Exporter popularised this by interpolating a time estimate from observed offset and timestamp samples, exposing kafka_consumergroup_group_max_lag_seconds alongside the offset-based kafka_consumergroup_group_lag. Worth knowing: that project’s repository has been archived and is read-only, so if you’re starting fresh, check whether your broker vendor or a maintained fork covers it before you deploy something unmaintained into the critical path.

Failure rate: decide what counts as a failure

Failure rate is trivially easy to compute and surprisingly easy to compute wrongly. The denominator and the definition both need a decision.

Count runs, not tasks. A DAG with sixty tasks where one flaps on a transient network error looks catastrophic at task level and fine at run level. The run is the unit the consumer cares about.

Count a run that succeeded on its third retry as a success for availability and a failure for a separate reliability metric. Both are true and they answer different questions. If your only metric folds retries into success, you will never see the slow degradation of an upstream API until it stops responding entirely.

sum by (pipeline) (rate(pipeline_runs_total{result="failure"}[6h]))
/
sum by (pipeline) (rate(pipeline_runs_total[6h]))

And add a category label for the failure reason at push time: source unavailable, schema mismatch, permission denied, timeout, validation failed. The rate tells you something is wrong. The category tells you who to wake up. Without it, every failure alert costs you a log dive before you can even route the incident.

Getting metrics out of a job that exits

Prometheus scrapes. Batch jobs finish and vanish. Pushgateway bridges that: the job pushes before exiting, and Pushgateway holds the values for Prometheus to scrape on its own schedule.

#!/usr/bin/env bash
set -euo pipefail

JOB="orders_load"
PGW="http://pushgateway.internal:9091"

start=$(date +%s)
if python /opt/pipelines/load_orders.py; then
  result=0
else
  result=1
fi
end=$(date +%s)

{
  cat <<EOF
# TYPE pipeline_run_duration_seconds gauge
pipeline_run_duration_seconds $((end - start))
# TYPE pipeline_last_run_timestamp_seconds gauge
pipeline_last_run_timestamp_seconds $end
# TYPE pipeline_last_run_success gauge
pipeline_last_run_success $((1 - result))
EOF
  if [ "$result" -eq 0 ]; then
    cat <<EOF
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $end
EOF
  fi
} | curl --fail --data-binary @- "$PGW/metrics/job/$JOB"

exit $result

The conditional block is doing real work. curl’s --data-binary issues a POST, and a POST to Pushgateway replaces only the metrics whose names appear in the payload, leaving the rest of the group intact. So a failing run updates the run timestamp and the success flag while leaving the previous pipeline_last_success_timestamp_seconds exactly where it was. That’s what lets time() - pipeline_last_success_timestamp_seconds keep climbing across consecutive failures. Send a PUT instead and you replace the whole group, wiping the value you needed.

On the Prometheus side, one setting is not optional:

scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['pushgateway.internal:9091']

Without honor_labels: true, Prometheus overwrites the job label your pipeline pushed with the scrape job’s own name, and every pipeline in your estate collapses into one indistinguishable series called pushgateway.

The trap worth internalising: Pushgateway never forgets. Metrics persist until something explicitly deletes them or the process restarts. A pipeline you decommissioned last quarter is still cheerfully reporting a success timestamp and a duration, and it looks alive on every dashboard. Treat Pushgateway as a cache of past executions, not a picture of current state, and use the push_time_seconds gauge that Pushgateway attaches to each group to tell the difference between a fresh push and a fossil.

If you’re on Airflow, the SLA feature is gone

This one will bite anyone upgrading. The sla and sla_miss_callback parameters were removed in Airflow 3.0, and the replacement, Deadline Alerts, arrived in 3.1. DAGs carrying the old configuration need manual migration; they don’t quietly keep working.

from datetime import timedelta

from airflow.sdk import AsyncCallback, DAG, DeadlineAlert, DeadlineReference
from airflow.providers.slack.notifications.slack_webhook import SlackWebhookNotifier

with DAG(
    dag_id="orders_load",
    deadline=DeadlineAlert(
        reference=DeadlineReference.DAGRUN_QUEUED_AT,
        interval=timedelta(minutes=45),
        callback=AsyncCallback(
            SlackWebhookNotifier,
            kwargs={"text": "orders_load has not finished 45 minutes after queuing."},
        ),
    ),
):
    ...

Note the reference point. Measuring from when the run was queued, rather than from its logical date, means scheduler backlog counts against the deadline. That’s usually what you want, because a run that sat in a queue for an hour is late to its consumers regardless of how fast it executed once it started.

There’s a structural weakness here worth naming: any alert that lives inside the orchestrator dies with the orchestrator. If the scheduler is down, nothing evaluates your deadline and nothing notifies anyone. The freshness probe from earlier is the answer, and it needs to run somewhere else. A small VPS from a provider like Contabo or InterServer running Prometheus and Alertmanager, or a hosted option like Grafana Cloud, gives you a watcher outside the blast radius of the thing being watched. This is the single highest-value piece of monitoring most data teams are missing.

Alerting without burning your on-call

The fastest way to make all of this worthless is to alert on every threshold crossing. Three rules keep it survivable.

1. Set thresholds from observed behaviour, not from wishes. Look at a month of actual freshness values and set warn_after a comfortable margin above the normal worst case. A source that habitually passes at eleven hours against a twelve-hour threshold is not healthy, it’s one upstream hiccup from paging you.
2. Page on consumer impact, not internal events. A task retry is not an incident. A dataset breaching the freshness commitment its consumers rely on is. Route everything else to a channel someone reads in the morning.
3. Use burn rate over multiple windows for SLOs. A short window catches fast breakage, a long window catches slow erosion, and requiring both to fire filters out the transient spikes that generate most false pages.

One more: alert on the absence of your own telemetry. A freshness gauge that stops updating is indistinguishable from one that’s fine, right up until someone asks about the numbers.

Troubleshooting

Freshness alert fires but the data looks current. Almost always a timezone problem. Your loaded_at column is in local time, now() is in UTC, and the offset shows up as a constant bias in the age. Store load timestamps in UTC and cast explicitly at read time.

Freshness flaps in and out of breach. Your check runs too close to the expected arrival. If loads land around six and your probe runs at five past, a fifteen-minute upstream delay produces an intermittent failure that trains everyone to ignore the alert. Move the probe later or widen the threshold.

Every pipeline reports as one series. honor_labels: true is missing from the Pushgateway scrape config.

A decommissioned pipeline still shows healthy. Stale group in Pushgateway. Delete the group and add a check on push_time_seconds so the next one surfaces on its own.

Latency looks fine but consumers say data is late. You’re measuring run duration and they’re feeling end-to-end. Add queue wait and event-time-to-load and the gap will be obvious.

Freshness passes, row count is near zero. The load timestamp advanced without meaningful data. This is a broken source or a stuck watermark, and it’s the reason the volume check belongs in the same probe.

Common mistakes

• Treating job success as the health signal and discovering staleness through a human.
• Running the freshness check inside the same pipeline it’s meant to police.
• Tracking freshness without volume, so a zero-row load reads as healthy.
• Reporting latency without saying which two timestamps it spans.
• Putting run IDs, batch IDs or timestamps into Prometheus labels, which multiplies your series count without bound. Those belong in traces or logs.
• Alerting on task-level failures instead of run-level outcomes, then muting the whole channel a week later.
• Assuming Pushgateway reflects current state rather than the last thing anyone pushed.

Best practices for data pipeline freshness monitoring

• Start with your three most-used tables. Full coverage is a project; three tables is an afternoon and catches most of the pain.
• Write down the freshness commitment per dataset in plain language, then encode it. “Yesterday’s orders are complete by 07:00” converts directly into a threshold.
• Name metrics consistently across pipelines. A shared prefix and a stable label set is what makes one dashboard work for all of them.
• Keep labels low-cardinality: pipeline, dataset, stage, environment, result. Nothing unbounded.
• Emit a failure category alongside every failure so alerts route themselves.
• Run the deadman check on infrastructure that doesn’t share a failure domain with the pipeline.
• Link every alert to a runbook that names the owner and the first three things to check.

Frequently asked questions

What is the difference between pipeline latency and data freshness?

Latency measures how long a specific batch of data took to travel from source to destination. Freshness measures how old the newest available record is right now, regardless of whether anything is currently running. A pipeline can have excellent latency and terrible freshness if it stopped being triggered.

How often should freshness checks run?

Frequently enough that you find out before your consumers do. A useful rule is roughly a quarter of your tolerance window: if data may be up to four hours old, check hourly. Freshness probes are cheap single-aggregate queries, so the limiting factor is usually warehouse billing rather than load.

Do I need a data observability platform for this?

Not to start. Freshness, latency and failure rate for a handful of critical datasets is a scheduled query, a push, and a few alert rules. Commercial platforms earn their cost when you need automatic column-level lineage, anomaly detection across hundreds of tables, or coverage of assets nobody has explicitly instrumented. That’s a real problem at scale, and a genuinely expensive one to build yourself. It’s just not the problem you have on day one.

How do I monitor freshness for a table that only updates weekly?

Set the threshold from the schedule plus a delivery margin, and add a deadman rule so a missing metric alerts on its own. For genuinely static reference tables, disable freshness explicitly rather than leaving a check that always warns; in dbt that means setting freshness to null for the table.

Should freshness checks fail the pipeline or just warn?

Both, in different places. Checking source freshness at the start of a job and failing hard prevents you building models on stale input, which is the cheapest bug to prevent and the most expensive to unwind. Checking output freshness after the fact should alert rather than fail, because the run is already over.

What percentile should I use for a latency SLO?

p95 for most internal analytics pipelines, p99 where downstream systems make automated decisions on the data. Track the median separately for capacity planning. Never use the average as the SLO number, because it hides exactly the tail that produces complaints.

Does this work for streaming pipelines too?

Yes, with different plumbing. Freshness becomes consumer lag measured in time, latency becomes event-time to availability-time, and failure rate becomes connector and task state plus dead-letter volume. The reasoning is identical; only the source of the numbers changes.

Conclusion

If you remember one thing, make it this: measure the data, not the job. Run status, task counts and duration all describe your code. Only freshness describes what your consumers actually receive, and it’s the one signal that stays honest when the pipeline succeeds at doing nothing.

Good data pipeline freshness monitoring is not a platform purchase. It’s a scheduled query that reports the age and volume of your most important tables, pushed somewhere durable, with a deadman rule so that silence is treated as a failure rather than as health. Add latency broken down by stage and failure rate categorised by reason, and you can answer “is the data good right now” without opening a single log file.

Need help instrumenting your pipelines?

Most of my consulting work in this area is retrofitting observability onto pipelines that already exist and can’t be paused. Specifically:

• Defining freshness and latency commitments per dataset, then translating them into thresholds and alert rules that hold up on-call.
• Adding freshness and volume probes to existing warehouses without touching the pipelines themselves.
• Wiring batch jobs into Prometheus through Pushgateway, including the grouping-key and stale-metric problems that bite six months later.
• Migrating Airflow DAGs off the removed SLA feature onto Deadline Alerts, or onto external checks that survive a scheduler outage.
• Building the Grafana dashboard that answers “is the data good right now” in one screen, with drill-down by stage and failure category.
• Cutting alert noise on pipelines where the channel has already been muted, by moving from task-level events to consumer-impact SLOs.

If you want a concrete starting point, send me a DAG file, a scrape config, or a screenshot of the dashboard you don’t trust, and I’ll tell you what I’d instrument first and why.

The post Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates appeared first on John Nessime.

That is the failure mode worth internalizing before anything else: Amazon Macie PII detection reports on what it managed to read, not on what is there. Objects it never opened produce no findings, and no findings looks identical to a clean result. The gap between “we scanned this” and “we found nothing” is where most Macie deployments quietly fail their first audit.

This post covers where that gap comes from and how to close it: the eligibility chain every object goes through, the four distinct reasons an object gets skipped, how to tune identifiers so the findings are worth reading, what actually drives the bill, and how to route results somewhere a human will act on them.

The eligibility chain behind Amazon Macie PII detection

Every object passes the same gates before a byte gets inspected. The order matters, because each gate fails differently and needs a different fix.

1. Is it in a general purpose S3 bucket? Directory buckets are out of scope entirely.
2. Is the storage class supported? Standard, Standard-IA, One Zone-IA, Intelligent-Tiering, Glacier Instant Retrieval and Reduced Redundancy are in. Glacier Deep Archive and S3 Express One Zone are not.
3. Does the object key carry a recognized file extension? This is a string check on the name, not a content sniff.
4. Can Macie retrieve and decrypt it, given the bucket policy, object ACL and encryption key?
5. Does the content parse, and stay inside the per-file quotas?
6. Do the active data identifiers match anything in it?

Only the last gate produces a finding. Everything before it produces silence. Macie does record why each object was skipped, but that lives in coverage data and object samples, not on the findings page most people look at.

One detail explains a lot of confusion: when you first enable automated sensitive data discovery, every bucket gets a sensitivity score of 50 and the label Not yet analyzed. A bucket whose permissions block Macie stays there permanently. It never turns red. It sits mid-list, looking unremarkable next to buckets that genuinely were analyzed.

Gap one: objects Macie never opened

Macie calls these unclassifiable. They fail the storage class or extension check, so no retrieval is attempted. This is the largest source of false confidence I’ve seen, and the cheapest to fix.

Missing file extensions

Classifiability comes from the file name extension. A file full of customer records named part-00003-a4f9 is invisible. The same bytes named part-00003-a4f9.json get inspected and produce findings.

This bites hardest on streaming ingestion. Amazon Data Firehose writes to S3 without appending an extension unless compression, format conversion, or the explicit file extension setting adds one. An uncompressed JSON stream lands as a tree of extensionless objects. Athena and Glue read them fine, because they infer format from the table definition. Macie skips every one.

Fix it upstream: set the S3 file extension on the delivery configuration, or enable GZIP compression or Parquet conversion, both of which append a recognized extension. If you can’t change the producer, the fallback is a copy-and-rename step into a scan prefix, which costs you storage plus a pipeline to maintain. Fixing the producer is almost always cheaper.

Unsupported storage classes

A lifecycle rule transitioning old exports to Glacier Deep Archive quietly removes them from scope. That may be the right storage decision, but it needs to be a conscious one, because archived customer data is still customer data when a regulator asks.

Macie exposes these counts per bucket, split by cause. This is the first query I run against a new account:

# Buckets where Macie can't classify objects, broken down by cause
aws macie2 describe-buckets 
  --query 'buckets[?unclassifiableObjectCount.total > `0`].{
      bucket: bucketName,
      classifiable: classifiableObjectCount,
      skipped_total: unclassifiableObjectCount.total,
      skipped_extension: unclassifiableObjectCount.fileType,
      skipped_storage_class: unclassifiableObjectCount.storageClass
    }' 
  --output table

Read the ratio, not the raw number. Forty thousand classifiable objects and twelve skipped is fine. Three classifiable and four hundred thousand skipped is a reporting artifact pretending to be a scan result, and that bucket’s sensitivity score means nothing.

Gap two: objects it opened and couldn’t read

These are classification errors rather than eligibility failures. Macie selected the object, tried to fetch it, and was refused. Three causes, and they need three different people to fix them.

• Customer-provided keys (SSE-C). Macie cannot supply the key material, so it cannot retrieve the object. No permissions fix exists. Re-encrypt with S3 managed or KMS keys if you want coverage.
• KMS key policy. For customer managed keys, the key policy must allow the Macie service-linked role, AWSServiceRoleForAmazonMacie, to decrypt. Cross-account buckets need the key owner to grant it, not the bucket owner. AWS publishes a permission analyzer script in the aws-samples/amazon-macie-scripts repository that enumerates every key Macie needs and generates the CLI commands to fix them.
• Restrictive bucket policies. An explicit Deny conditioned on source IP or VPC endpoint blocks Macie along with everything else. The working pattern excludes the service-linked role ARN from the deny using the aws:PrincipalArn condition key.

To see which objects Macie actually touched in a bucket, pull the object samples. It’s the closest thing to a scan log you get:

# Objects automated discovery selected, with per-object status
aws macie2 list-resource-profile-artifacts 
  --resource-arn arn:aws:s3:::your-bucket-name

Anything with a status of SKIPPED is a lead. Take the key, check its encryption settings in S3, and you usually have your answer inside a minute.

Gap three: it read the file and nothing matched

Macie ships managed data identifiers covering common PII, financial data and credentials across many countries. Automated discovery uses a recommended subset by default rather than all of them. Sensible, but it is a default, and it will miss things that matter to you.

Add the managed identifiers for your actual jurisdictions. If you hold records for customers in a country whose national ID identifier isn’t in the recommended set, you’re scanning for the wrong things with perfect efficiency. Pull the current list with aws macie2 list-managed-data-identifiers and compare it against where your customers live.

Write custom identifiers for what only you know is sensitive. Internal account numbers, case references, employee IDs. A custom identifier is a regex plus optional keywords and a proximity rule, and the keywords are what save you. A bare [0-9]{8} matches timestamps, order totals and row counts, and you drown. The same regex with keywords and a match distance only fires when the number sits near a word that gives it meaning.

Test the pattern before it goes near a job. This runs the criteria against sample text and returns match counts without creating anything:

# Dry-run a custom identifier before creating it
aws macie2 test-custom-data-identifier 
  --regex 'ACC-[0-9]{4}-[0-9]{4}' 
  --keywords account customer acct 
  --maximum-match-distance 30 
  --sample-text 'customer account ACC-4821-9930 updated; total 1234-5678'

Run it against real samples, including the awkward ones, then against a file you know is clean and confirm zero. A pattern you only tested on positives is a pattern you haven’t tested.

The mirror image is allow lists, which tell Macie to ignore specific text or patterns: your support inbox address, your published press contact, the seeded test records in everyone’s staging fixtures. Without them the same non-issues appear in every scan until the team stops reading findings. Allow lists accept predefined text or regex, and the per-account and per-job quotas are tight enough that you should curate rather than dump.

Gap four: quota truncation inside large files

Subtle, because the object is analyzed. Just not completely, and a partial result reports like a full one.

• Per-file size quotas differ by format. Exceed the quota for a type and Macie analyzes none of that file, not part of it.
• Archives have separate ceilings for nesting depth, extracted bytes and file count. If the metadata shows a breach up front, nothing is extracted. Cross a limit partway through and Macie stops, reporting only what it processed.
• JSON and JSON Lines have a nested depth limit. Past it, the file is not analyzed at all.
• Full names and mailing addresses cap out per file. After the cap Macie stops counting and stops reporting locations for that type, so the occurrence count is a floor, never a total.

Findings carry a status reason naming the limit that was hit, with values like FILE_EXCEEDS_SIZE_LIMIT, ARCHIVE_NESTING_LEVEL_OVER_LIMIT and JSON_NESTING_LEVEL_OVER_LIMIT. Filter on those before declaring a data lake of large compressed exports clean. Check current quota values in the Macie documentation rather than memorizing them, since they differ per format.

Automated discovery or a discovery job?

Automated sensitive data discovery evaluates your inventory daily and samples representative objects across as many buckets as it can, grouping by metadata like prefix, extension and last-modified date. It is breadth-first and it does not re-analyze unchanged objects. Its job is to tell you which buckets deserve attention. It is not designed to prove a bucket is clean, and reading it that way is the root of most Macie misunderstandings.

Sensitive data discovery jobs are the depth tool. You choose the buckets, scope by prefix, extension, size or object tag, choose the identifiers, and run once or on a schedule. This is what you point at the bucket automated discovery just flagged.

# One-time job over a single prefix, recommended identifiers, 30% sampling
aws macie2 create-classification-job 
  --job-type ONE_TIME 
  --name "exports-prefix-review" 
  --managed-data-identifier-selector RECOMMENDED 
  --sampling-percentage 30 
  --s3-job-definition '{
    "bucketDefinitions": [
      {"accountId": "111122223333", "buckets": ["your-bucket-name"]}
    ],
    "scoping": {
      "includes": {
        "and": [
          {"simpleScopeTerm": {
             "comparator": "STARTS_WITH",
             "key": "OBJECT_KEY",
             "values": ["exports/"]
          }}
        ]
      }
    }
  }'

One detail about --sampling-percentage that trips people up: it selects a random percentage of eligible objects and then analyzes each selected object completely. It does not read a fraction of each file. For a bucket where every object comes from one pipeline, 20 or 30 percent tells you what you need. For heterogeneous uploads, sampling is a coin flip and you want the full pass.

What actually drives the bill

Macie bills on three dimensions, and the one that surprises people is not the one they budget for.

• Bucket evaluation. Per general purpose bucket monitored, prorated daily, up to the account ceiling. Predictable, driven by bucket count rather than data volume.
• Object monitoring. Per object tracked while automated discovery is enabled. This is inventory bookkeeping, not content reading, and it runs whether or not anything gets scanned that day. On an account with a log archive holding hundreds of millions of tiny objects, it can dwarf the other two.
• Data inspected. Per GB actually analyzed, covering both automated discovery and jobs.

So excluding buckets from automated discovery is a genuine cost lever, not just noise reduction. Log archives, CloudTrail destinations, build artifact stores and backup targets are usually safe exclusions. Exclude for the right reason though: a backup bucket holding database dumps of your customer table is exactly what you want scanned, however boring its name sounds.

The three dimensions appear as distinct usage types in the Cost and Usage Report, so they split cleanly. A FinOps platform like Vantage or CloudZero, or just a Cost Explorer view grouped by usage type, tells you within a day whether object monitoring or inspection is driving the number. That changes the fix entirely: object monitoring is solved by scoping, inspection by sampling and scheduling.

Getting findings somewhere a human sees them

Findings that only exist in the Macie console get read during onboarding week and never again. Two things are worth wiring up on day one.

Publish findings outward. Macie emits findings to EventBridge and integrates with Security Hub. EventBridge earns its keep because you can filter on severity and finding type and route only what matters, whether that’s a Slack channel, a ticket, or a Lambda that tags the bucket. If you already centralize alerts in Grafana Cloud or similar, that path keeps sensitive data findings next to everything else on call sees.

Configure the discovery results repository. Findings tell you what was found and where. The detailed discovery results, including analysis logs for objects where nothing was found, need an S3 bucket and a KMS key configured to persist. Nothing is retained long term until you set this up, and “we scanned it and found nothing” is exactly the record an auditor asks for.

One operational note on the reveal feature, which retrieves sample occurrences so a human can confirm a finding is real. It needs a customer managed KMS key and it’s genuinely useful for triage. It also means someone is now looking at live customer PII, which belongs in your access model and audit trail. Same goes if an engineer downloads an object to investigate locally: that copy is unmanaged PII on a laptop and needs a secure deletion step, whether that’s a wipe tool from a vendor like O&O Software or a documented and enforced process. Findings triage has a habit of creating the exposure it was meant to prevent.

Troubleshooting a bucket that reports nothing

When Amazon Macie PII detection reports nothing for a bucket you have doubts about, work down this in order. Each step rules out one gate, and stopping at the first hit saves you the rest.

1. Check the sensitivity label. Not yet analyzed at score 50 means nothing was analyzed. Do not read that as clean.
2. Check the coverage page. Macie names the issue directly: access denied, invalid encryption, invalid KMS key, permission denied, or unclassifiable.
3. Compare classifiable and unclassifiable counts with the describe-buckets query above.
4. List a few object keys. Run aws s3 ls against a prefix and look for extensions. Extensionless keys explain the whole thing.
5. Check the storage class. A lifecycle rule may have moved everything somewhere Macie doesn’t read.
6. Check encryption. SSE-C is unfixable without re-encryption. A customer managed KMS key is fixable with a key policy change.
7. Check object samples with list-resource-profile-artifacts to see exactly what was selected and skipped.
8. Only now question the identifiers. Run a targeted job with a custom identifier for the data you expect and see whether it fires.

Common mistakes

• Treating a low sensitivity score as evidence of no PII. It reflects what was found and how much was analyzed, so low coverage produces a low score.
• Enabling Macie in one region and assuming account-wide coverage. Settings and results are per region, every time.
• Running default identifiers in a business operating outside the jurisdictions the recommended set covers.
• Building custom identifiers from regex alone, then abandoning Macie over the false positive volume.
• Skipping the discovery results repository, then having no evidence trail when someone asks what was scanned and when.
• Excluding buckets by name pattern without checking contents. Backup and export buckets are frequently the highest-risk ones you own.

Best practices

• Measure coverage before findings. A coverage report is the first deliverable of a Macie rollout, not an afterthought.
• Fix extensions at the producer rather than building a rename pipeline downstream.
• Run the KMS permission analyzer script once per account, then again whenever a new customer managed key appears.
• Use automated discovery for breadth and targeted jobs for depth, letting the first choose targets for the second.
• Test every custom identifier against both a positive and a negative sample before it reaches a job.
• Curate allow lists early. The cost of a noisy first month is a team that stops reading findings.
• Re-check coverage after any change to bucket policies, KMS keys, lifecycle rules or ingestion pipelines. All four silently remove data from scope.

Frequently asked questions

Can Amazon Macie scan anything other than S3?

No. Macie analyzes objects in S3 general purpose buckets only. The documented workaround is bringing data to it: export RDS or Aurora snapshots to S3 in Parquet, or export a DynamoDB table to S3, then run a discovery job against the export. That works, but the export is now a second copy of your sensitive data with its own encryption and deletion requirements.

Does Macie scan new objects as they land?

Not per object. Automated discovery evaluates inventory on a daily cycle and prioritizes objects that are new or recently changed, so fresh data moves up the queue rather than triggering an immediate scan. If you need scanning tied to arrival, drive a discovery job from S3 event notifications through EventBridge and accept that you own that orchestration.

Why does a bucket I know has PII show a low sensitivity score?

Almost always coverage rather than detection. Check whether the objects are classifiable, whether Macie can decrypt them, and whether the label reads Not yet analyzed. If Macie genuinely read them and found nothing, the next suspect is the identifier set, and a targeted job with a custom identifier will tell you in one run.

Do I need both automated discovery and discovery jobs?

For most teams, yes. Automated discovery gives you the map at predictable cost. Jobs give you proof for a specific bucket at a specific time, which is what compliance evidence actually requires. Running only jobs means you never discover the bucket nobody told you about.

How do I cut false positives without missing real PII?

In order of preference: add keywords and a proximity rule so matches need context; use allow lists for specific known-benign values like your published contact details and test fixtures; and only then remove managed identifiers that are consistently wrong for your data. Removing identifiers is the bluntest option and the easiest to regret, so leave it last.

Does Macie work across accounts in an organization?

Yes, through AWS Organizations with a delegated administrator. Design around this: the administrator’s automated discovery settings apply to member accounts, members can’t change them, and members see coverage and sensitivity data for their own buckets but not the sensitive data findings themselves. That shapes who can actually remediate what.

The one thing worth remembering

Amazon Macie PII detection is only as good as its coverage, and coverage fails silently by design. An object with the wrong extension, the wrong storage class, or an unreachable encryption key produces no finding, and no finding renders identically to a clean result. Before you hand a Macie report to anyone who will make a decision from it, put the coverage numbers next to it: how many objects were classifiable, how many were analyzed, how many were skipped and why.

Do that once and the rest of the service becomes straightforward. Skip it and you’re maintaining an expensive dashboard reporting on whichever subset of your data happened to be readable.

Need help getting Macie to actually see your data?

Most of the work in a Macie rollout isn’t turning it on. It’s the unglamorous part: proving what got scanned, fixing the reasons things didn’t, and making the output land somewhere a human acts on it. That’s the part I help with.

• Coverage audits across your S3 estate, with per-bucket classifiable, analyzed and skipped counts and the specific cause of each gap
• Fixing those causes: KMS key policies, bucket policies blocking the service-linked role, lifecycle rules archiving data out of scope, pipelines writing extensionless objects
• Custom data identifiers and allow lists tuned against your real data and tested on positive and negative samples first
• Cost shaping: working out whether object monitoring or data inspection drives your bill, and scoping discovery so the number is defensible
• Findings pipelines through EventBridge or Security Hub into the alerting and ticketing you already run, with severity filtering that keeps signal above noise
• Multi-account setup under AWS Organizations, including the delegated administrator model and who can see and remediate what

If you want a second opinion, send me the output of the describe-buckets query above, or a screenshot of your coverage page, and I’ll tell you what’s actually being scanned.

The post Amazon Macie PII Detection: The Buckets It Never Opened appeared first on John Nessime.

That failure should shape how you think about EKS Pod Identity vs IRSA. Both deliver the same end result to your application: short-lived, rotated credentials with nothing hardcoded. The differences are in how trust is expressed, who has to touch IAM to change it, and what happens when the wiring is subtly wrong.

This post compares the two on what actually decides the choice: how each scales across clusters, where each refuses to work at all, the failures each hides, and a decision procedure you can apply to a real cluster.

Same outcome, two very different trust shapes

Most of the trade-offs fall out of the mechanism, so it is worth being precise about both.

IRSA: federation through an OIDC provider

The cluster has an OIDC issuer, which you register as an identity provider in IAM. A projected service account token is mounted into the pod, and the SDK exchanges it for credentials through the web identity flow against STS. The role’s trust policy is what makes that work, and it names one specific cluster:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub":
          "system:serviceaccount:production:my-app",
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud":
          "sts.amazonaws.com"
      }
    }
  }]
}

The important detail is that the issuer URL is unique per cluster. The relationship between a workload and a role lives inside IAM, and it is cluster-specific by construction.

Pod Identity: an EKS API and an on-node agent

Pod Identity moves that mapping out of IAM. You install the EKS Pod Identity Agent add-on, which runs as a DaemonSet, then create an association through the EKS API binding a namespace plus service account name to a role. The agent calls the EKS Auth API for temporary credentials, and the pod picks them up through the container credential provider that AWS SDKs already support.

The trust policy contains nothing cluster-specific:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
    "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": [ "sts:AssumeRole", "sts:TagSession" ]
  }]
}

sts:TagSession is not decoration. Pod Identity attaches session tags on every assume, so a policy allowing only sts:AssumeRole fails. It is the most common first-attempt error.

Creating the association is a single call:

aws eks create-pod-identity-association 
  --cluster-name prod-eu 
  --namespace production 
  --service-account my-app 
  --role-arn arn:aws:iam::111122223333:role/my-app

Where IRSA still wins

Treating IRSA as legacy is the most common mistake here. For some workloads it is not the older option, it is the only option.

• Fargate pods. The Pod Identity Agent is a DaemonSet on EC2 nodes. Fargate has no node for it to land on, and AWS documents Fargate pods as unsupported. If part of your workload runs on Fargate, IRSA is the mechanism there, permanently, not as a stopgap.
• Windows nodes. Also documented as unsupported for Pod Identity.
• Older SDKs you cannot rebuild. The web identity flow has been in the SDKs far longer, so a vendor image you cannot patch is more likely to work with IRSA.
• Direct federation into workload accounts. If your model wants the pod’s identity federating straight into a role in the resource-owning account, IRSA does that natively. Pod Identity reaches other accounts by role chaining, a different trust shape even when the outcome looks the same.
• You already have it working at a scale you control. A few clusters with a Terraform module that provisions the provider and the roles is not a problem that needs solving.

Where IRSA hurts

The pain is administrative, and it grows with cluster count rather than workload count. Every new cluster brings a new OIDC issuer, which means editing the trust policy of every role those workloads use. Trust policies have a size limit, so a role trusting many issuers eventually runs out of room and you end up duplicating roles to work around it.

The same asymmetry shows up in blue/green cluster upgrades. Standing a new cluster beside the old one means every role used by every migrating workload needs a trust policy edit before the new cluster can do anything, and another edit afterwards to clean up. That is IAM work on the critical path of an upgrade, usually owned by a different team than the one doing it.

Where Pod Identity wins

• Roles become portable. One role, one generic trust policy, any number of clusters. Adding a cluster is an EKS API call per workload, not an IAM edit per role.
• The permission split matches how teams are organised. Creating an association needs EKS permissions, not iam:UpdateAssumeRolePolicy, so cluster admins stop filing IAM tickets for every new service account.
• Session tags arrive for free. Every assume carries eks-cluster-arn, eks-cluster-name, kubernetes-namespace, kubernetes-service-account, kubernetes-pod-name and kubernetes-pod-uid. That is usable attribute-based access control and the CloudTrail audit trail you always wanted.
• Cross-account access without touching application code. An association can carry a target role ARN, and EKS performs the role chaining for you.

The ABAC angle is underrated. One policy can serve dozens of workloads, because the session tag identifies the caller:

"Condition": {
  "StringEquals": {
    "s3:ExistingObjectTag/team": "${aws:PrincipalTag/kubernetes-namespace}"
  }
}

Where Pod Identity hurts

The generic trust policy is a real trade-off. Written exactly as AWS shows it, that role can be assumed on behalf of any service account in any namespace in any cluster in the account, as long as someone with EKS permissions creates an association pointing at it. Blast radius control has moved from IAM into the EKS API. If those permissions are loosely held, you widened it.

The fix is to put the constraint back in the trust policy using the request tags Pod Identity sends:

"Condition": {
  "StringEquals": {
    "aws:RequestTag/kubernetes-namespace": "production",
    "aws:RequestTag/kubernetes-service-account": "my-app"
  }
}

You can also pin to a cluster with aws:SourceArn. That costs some of the portability you switched for, which is the point: decide per role. A role reading one team’s bucket should be pinned. A role used by an identical workload in six clusters should not.

The other cost is a new moving part. The agent is a DaemonSet, so it needs to tolerate your taints, and it must be running before pods on that node can get credentials.

The failures that actually bite

Silent fallback to the node role

This is the one from the opening, and it is invisible by design. Pod Identity works through the container credential provider, which the SDK finds by reading two environment variables EKS injects into the pod:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

An SDK that predates that provider does not error on these. It does not know what they mean, so it keeps walking the credential chain until it reaches instance metadata and picks up the node role. The application works. The permissions are wrong. Nothing complains.

Two defences, and you want both. Check the minimum SDK version list in the EKS documentation and pin base images above it, and block pod access to IMDS so falling back has nowhere to fall. The first prevents the problem; the second turns a silent success into a loud failure.

Verify from inside the pod. The ARN in the response tells you which identity is really in use:

kubectl exec -n production deploy/my-app -- aws sts get-caller-identity

The association that matches nothing

An association is an exact match on cluster, namespace and service account name. A typo in any of the three produces an association that exists, looks fine in the console, and never applies to a pod. List them and read them against your manifests:

aws eks list-pod-identity-associations --cluster-name prod-eu

Associations also only take effect for pods started after they exist. If you create one and nothing changes, restart the workload before you start debugging IAM.

Both mechanisms configured at once

During a migration you will often have a service account carrying an IRSA annotation and a Pod Identity association at the same time. Both paths can work, and which one the SDK uses depends on credential chain ordering. AWS documents that credentials found earlier in the chain keep being used even when an association exists.

Do not reason about it. Check the caller identity and check CloudTrail, then remove the one you are not keeping so nobody has to work it out again.

PackedPolicyTooLarge

Session tags, managed policy ARNs and inline session policies are compressed into a packed format with its own size limit, and six session tags on every assume is not free. The association can disable the automatic tags, at the cost of the ABAC and audit benefits that were half the reason to use Pod Identity.

A decision procedure for EKS Pod Identity vs IRSA

Work through these in order and stop at the first one that answers.

1. Does this workload run on Fargate or Windows nodes? IRSA. There is no decision to make.
2. Can you rebuild the image with a current SDK? If not, IRSA, or fix the image first.
3. Does your model require direct OIDC federation into the resource account? IRSA. Role chaining is not the same shape, even if it reaches the same bucket.
4. Will this role be used from more than one cluster, now or after the next upgrade? Pod Identity. The saving compounds every time you add or replace a cluster.
5. Do you want cluster and namespace in the CloudTrail record without extra work? Pod Identity, because of the session tags.
6. None of the above? Pod Identity for anything new, IRSA left alone where it already works.

AWS’s own guidance lands in the same place: Pod Identity for new applications on supported node types, IRSA where you already have it working or where Pod Identity cannot run.

Arguments that don’t survive contact

• “IRSA is deprecated.” It is not. AWS has said it continues to invest in it, and Fargate support alone guarantees it stays.
• “Pod Identity is less secure because the trust policy is generic.” Only if you leave it generic. Request tag conditions scope it as tightly as an IRSA trust policy, and you keep the session tags IRSA does not give you.
• “Run one mechanism per cluster, cleanly.” A mixed cluster is documented, supported, and for clusters with Fargate pods it is the permanent steady state, not a transitional mess.
• “EKS add-ons can only use IRSA.” True early on, and still repeated in older write-ups. Add-on creation and update now accept Pod Identity associations directly. Check your specific add-on rather than trusting a blog post, including this one.
• “Migrate everything, then delete the OIDC provider.” Deleting it breaks every role still trusting it, including ones you forgot. Migrate workload by workload, and delete the provider only after CloudTrail shows zero AssumeRoleWithWebIdentity calls for that issuer across a full batch cycle.

How I’d decide, and what I’d do first

• Default new workloads on EC2 or Karpenter-managed nodes to Pod Identity, and leave working IRSA alone until you have a reason to touch it.
• Add request tag conditions for anything sensitive. Portability helps a shared platform role and hurts a role that reads customer data.
• Restrict pod access to IMDS before you migrate anything, not after. It converts the silent failure into a visible one.
• Migrate with the role trusting both principals at once, so rollback is a manifest change rather than an IAM change.
• Put credential identity in your dashboards. A CloudTrail query counting calls made by node roles from workload namespaces catches the exact failure this post opened with. Amazon Managed Grafana or Grafana Cloud will render it if you already ship CloudTrail somewhere; a scheduled CLI check from a CI runner or a small VPS from Contabo or InterServer is a fine low-tech alternative.
• Codify associations in Terraform or OpenTofu from the start. They are cheap to create by hand and easy to lose track of.

Frequently asked questions

Is EKS Pod Identity replacing IRSA?

No. AWS recommends Pod Identity for new applications on supported node types and continues to support IRSA. Since Pod Identity cannot run on Fargate or Windows nodes, IRSA stays for the workloads that live there.

Can I use both in the same cluster?

Yes, and it is a documented, supported pattern. Older workloads on IRSA and new ones on Pod Identity is normal. Avoid configuring both for the same service account beyond a migration window, because which one wins depends on SDK credential chain ordering.

Why does my pod get AccessDenied after switching to Pod Identity?

Check the trust policy for sts:TagSession first. Pod Identity always sends session tags, so a trust policy allowing only sts:AssumeRole is rejected. After that, check that the namespace and service account in the association match the pod exactly.

Does Pod Identity work for cross-account access?

Yes, through role chaining. The association takes a role in the cluster account plus a target role in the resource account, and EKS chains them. It also exposes an external ID you can require in the target role’s trust policy to prevent a confused deputy problem.

Do I have to change my application code?

No, provided the SDK is recent enough. Both work through the default credential chain, so the application calls AWS as usual. The only change that bites is an SDK too old to understand the container credential provider.

How do I confirm which identity a pod is actually using?

Run aws sts get-caller-identity inside the container and read the assumed role ARN. If it names the node role rather than the one you assigned, the chain fell through. Cross-check in CloudTrail, where session tags also give you the namespace and service account for Pod Identity sessions.

The one thing worth remembering

The real question in EKS Pod Identity vs IRSA is not which is more modern. It is where you want the cluster-to-role relationship to live. IRSA puts it in IAM trust policies: precise, cluster-specific, and increasing administrative drag as clusters multiply. Pod Identity puts it in the EKS API: portable, fast to change, and it hands you the job of deciding per role how much of that portability you actually want.

Pick per workload, not per cluster. And whichever you pick, verify the identity from inside the pod, because the worst outcome here is not a permission error. It is a workload that has been running with the wrong credentials since the day you deployed it.

Need a second pair of eyes on your EKS workload identity?

Most of this work is unglamorous: reading trust policies, matching associations against manifests, proving what a pod is really authenticating as. Things I help with:

• Auditing every service account in a cluster and reporting which identity its pods actually use, not the one the manifest claims
• Planning and running an IRSA to Pod Identity migration workload by workload, with a rollback that needs no IAM change
• Writing scoped trust policies with request tag conditions, so portability is a deliberate choice per role
• Restricting pod access to IMDS and confirming nothing was silently depending on the node role
• Setting up cross-account access with target roles and external IDs, plus the CloudTrail view that proves it is used as intended
• Codifying associations and roles in Terraform or OpenTofu, so the next cluster is a variable change rather than a project

Send me a trust policy, a service account manifest, or the output of aws sts get-caller-identity from inside one of your pods, and I will tell you what it is really doing.

The post EKS Pod Identity vs IRSA: What Actually Decides the Choice appeared first on John Nessime.

That’s the signature failure of telemetry pipelines on AWS. Transport worked, authentication worked, the endpoint answered 200. It kept part of the payload and dropped the rest, because something in the batch broke a limit your collector has no idea exists.

Getting started with OpenTelemetry on AWS is not the hard part. AWS Distro for OpenTelemetry (ADOT) will have you shipping traces in an afternoon. The hard part is four decisions underneath it: where the collector sits, which door telemetry uses to get into AWS, what happens to your metrics on the way through, and whether a second backend later costs you a config line or a migration. This post covers those four decisions and how each one fails without announcing itself.

What ADOT actually buys you

ADOT is not a separate protocol or agent. It’s a downstream build of the upstream OpenTelemetry Collector plus AWS-flavoured SDKs, tested by AWS and covered by AWS Support. The config syntax is the upstream syntax, so a config written for vanilla OTel runs on ADOT and vice versa. The difference is which components are compiled in: ADOT ships the X-Ray exporter, the CloudWatch EMF exporter, the SigV4 extension, and ECS metric receivers already there.

So the choice is narrower than it looks. Take ADOT for a build someone else validates and supports. Take upstream Contrib if you need a component ADOT hasn’t bundled, or you run one collector image across AWS, another cloud, and bare metal. Neither choice touches your instrumentation, which is the entire point.

Decision one: where the collector runs

Three shapes, trading the same three things: blast radius, cost, and how much processing happens before data leaves your network.

Sidecar

One collector container per ECS task or pod. The application talks to localhost, so no service discovery and no network hop to get wrong, and failure stays contained to one workload. The cost: you pay for that container everywhere, batches stay small because batching is per-instance, and any config change is a redeploy of every task.

Agent plus gateway

A light collector per node (a DaemonSet on EKS) forwarding to a small pool of gateways. This is the shape I reach for first on anything past a handful of services. The agent does host-level enrichment, the gateway does the expensive work: large batches, tail sampling, redaction, fan-out. Config changes hit the gateway only.

The catch: a gateway pool is a thing you now operate. It needs autoscaling and its own alerting, and undersized it starts refusing data at exactly the moment you have an incident and volume spikes.

No collector at all

The ADOT SDKs can export straight to the CloudWatch OTLP endpoints, signing requests with credentials already available to the process. For a handful of Lambda functions this is the right answer: nothing to run, nothing to scale, no extra container in the cold start path. What you give up is control. Batching, retries, sampling policy, redaction, and backend routing all move into application processes and their environment variables, so changing any of them is a fleet redeploy rather than a config push. Good starting point, poor steady state.

Decision two: which door into AWS

Two ways to hand telemetry to AWS, and they behave very differently.

The older path uses AWS-specific exporters. awsxray converts OTLP spans into X-Ray segment documents and calls the X-Ray API. awsemf converts OTLP metrics into CloudWatch Embedded Metric Format and writes them as log events. Both work, both are well trodden, and both reshape your data into an AWS-native format on the way out.

The newer path is native OTLP. CloudWatch exposes OTLP endpoints per signal, reached with the plain otlphttp exporter and a SigV4 signer. Same exporter you’d point at Grafana Cloud or Honeycomb, different URL and authenticator. That’s what “export anywhere” means in practice, and it’s why I default to it on new work.

The endpoints follow a per-service pattern:

• Traces: https://xray.<region>.amazonaws.com/v1/traces
• Metrics: https://monitoring.<region>.amazonaws.com/v1/metrics
• Logs: https://logs.<region>.amazonaws.com/v1/logs

Three constraints there will each cost you an afternoon if you don’t know them going in.

HTTP only. No gRPC. If your collector exports over otlp on 4317 you can’t just change the URL, you have to switch exporters. Receiving gRPC from applications is fine, it’s the outbound leg that must be HTTP.

SigV4 required. That means the sigv4auth extension, configured per signal because the signing service name differs: xray, monitoring, and logs respectively. Bearer tokens are an alternative for metrics and logs, but not for traces.

Logs need headers, not just a URL. Target log group and stream travel in x-aws-log-group and x-aws-log-stream headers. Omit them and the request has nowhere to land.

A working shape for traces and logs looks like this:

extensions:
  sigv4auth/traces:
    region: "us-east-1"
    service: "xray"
  sigv4auth/logs:
    region: "us-east-1"
    service: "logs"

exporters:
  otlphttp/traces:
    compression: gzip
    traces_endpoint: https://xray.us-east-1.amazonaws.com/v1/traces
    auth:
      authenticator: sigv4auth/traces

  otlphttp/logs:
    compression: gzip
    logs_endpoint: https://logs.us-east-1.amazonaws.com/v1/logs
    headers:
      x-aws-log-group: MyApplicationLogs
      x-aws-log-stream: default
    auth:
      authenticator: sigv4auth/logs

service:
  extensions: [sigv4auth/traces, sigv4auth/logs]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/traces]
    logs:
      receivers: [otlp]
      exporters: [otlphttp/logs]

Note traces_endpoint and logs_endpoint rather than endpoint. This one bites people. The endpoint setting is a base URL and the exporter appends the signal path itself, so endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics actually requests /v1/metrics/v1/metrics. The per-signal settings take a full path. If you’re getting 404s from a URL you’re certain is right, this is why.

Also worth a comment in your config: upstream has renamed the component to otlp_http and otlphttp is now a deprecated alias scheduled for removal. Both work today.

Traces need Transaction Search turned on first

This is the single most common reason a correctly configured trace pipeline produces nothing. The X-Ray OTLP endpoint requires Transaction Search to be enabled on the account, which redirects span ingestion into CloudWatch Logs:

aws xray update-trace-segment-destination --destination CloudWatchLogs

Spans then land in a log group named aws/spans, with a percentage indexed in X-Ray as trace summaries for search. The default index rate is one percent, enough to find traces while all spans stay queryable as structured logs. The caller needs xray:UpdateTraceSegmentDestination and xray:UpdateIndexingRule plus log group creation rights.

Two consequences to plan for. Span ingestion is billed separately from log ingestion, so it’s a line item rather than a rounding error. And AWS recommends always_on sampling in the SDK on this path, because the indexing rule already handles reduction. Sample in both places and your service map goes patchy.

The X-Ray SDK clock is running

The dates matter here, so plainly: the X-Ray SDKs and daemon entered maintenance mode on 25 February 2026 (security fixes only, no new instrumentation support), with end of support on 25 February 2027. The X-Ray service is fine and still gaining features. It’s the client libraries and the UDP daemon winding down. If you run aws-xray-sdk and a daemon sidecar, that is technical debt with a published expiry, and either the collector or the CloudWatch agent replaces the daemon.

Decision three: metrics, temporality, and the cardinality bill

Metrics are where the silent drops live, because collector defaults and endpoint limits disagree.

The metrics endpoint caps a single request at 1 MB uncompressed and 1,000 datapoints, counted across resource, scope, and metric levels combined. The upstream batch processor’s default is far larger. Leave it alone and you build oversized requests, and the response to an oversized or partially invalid request is not always a clean failure: it can come back 200 with some metrics accepted and others rejected or throttled. So set the batch size deliberately. AWS’s own examples use a conservative value:

processors:
  batch:
    send_batch_size: 200
    timeout: 10s

The other limits shape attribute design more than config:

• 150 labels maximum across resource, scope, and datapoint attributes per datapoint
• 40 KB combined label and value size per series per datapoint
• One million new series creatable per ten-minute window, per account
• Timestamps no more than ten minutes in the future or fourteen days in the past

That new-series ceiling catches teams out. Attach a request ID, a session ID, or a raw URL path to a metric attribute and every request mints a fresh series. You’ll hit a million faster than you expect, and it presents as metrics randomly going missing rather than a quota error. Unbounded identifiers belong on spans and logs. Metric attributes should be values you could enumerate on a whiteboard.

Temporality is the other conscious decision. The OpenTelemetry SDK spec defaults to cumulative, while CloudWatch’s metric model is delta-shaped: it wants what happened this period, not the total since your process booted. Counters that look like ever-climbing staircases instead of rates are that mismatch. The cumulativetodelta processor converts in the pipeline, keeping the decision in the collector rather than scattered across SDK environment variables in every service.

Decide once and centrally either way. Mixed temporality across one account produces dashboards that are subtly wrong for months before anyone notices. And once volume climbs, pointing a cost tool such as Vantage or CloudZero at the CloudWatch line items earns its setup time, because telemetry spend grows in steps nobody approved.

Decision four: a second backend without a second instrumentation

Here’s the payoff. A collector pipeline holds multiple exporters, and adding one is a config change:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/traces, otlphttp/vendor]

The same spans now reach CloudWatch and Grafana Cloud, Honeycomb, Datadog, or a self-hosted Tempo, with no application redeploy and no second agent. That’s the cashable value of instrumenting once against an open protocol: the switching cost of a backend drops from a migration project to a pull request.

Be honest about the price. You pay ingest twice, and egress from the gateway is real money at volume. Collector memory scales with exporter queue count, so a fan-out gateway needs headroom. And if one backend slows, backpressure can reach the pipeline feeding the other, which argues for separate pipelines per destination when their reliability differs.

It works in reverse too. Because the metrics and logs endpoints accept bearer tokens, a machine with no AWS credentials at all can ship into CloudWatch: a CI runner, another cloud, or a VPS at a provider like Contabo or InterServer running part of your stack. One collector config, one destination, regardless of who owns the hardware. Traces are the exception and still require SigV4.

Why OpenTelemetry on AWS fails quietly

Most breakages here don’t throw. How to recognise the common ones:

Data is missing but nothing errors

Almost always a limit breach inside an accepted request. Check batch sizes against the per-signal caps first, then attribute counts. Turn on the collector’s internal telemetry and compare the exporter’s sent counters against its failed counters. If sent looks healthy and data is still missing, the loss is happening server-side after acceptance, which narrows it to limits.

403 with a signature mismatch

A SigV4 problem, not an IAM problem. The usual cause is the wrong signing service name in the extension, since xray, monitoring, and logs are not interchangeable. If you sign requests yourself rather than using the extension, note that the traces endpoint is stricter about which headers land in the signed set, so sign a minimal stable set rather than everything the HTTP client added.

Backfilled data vanishes

Every endpoint enforces a timestamp window, and fourteen days in the past is the outer edge for all three signals. Replaying an old queue past that boundary gets rejected. Not a bug, but it looks like one when the replay job appears to succeed and produces nothing.

Traces arrive but the service map is empty

Usually a missing or inconsistent service.name resource attribute, or double sampling between the SDK and the indexing rule. Confirm what’s actually leaving the collector before hunting in the console: add the debug exporter to a copy of the pipeline, set verbosity: detailed, and read the resource attributes on real spans.

Common mistakes

• Pointing an OTLP gRPC exporter at a CloudWatch endpoint. They’re HTTP only, and the error won’t say so clearly.
• Using endpoint with a full signal path instead of the per-signal traces_endpoint, metrics_endpoint, or logs_endpoint.
• Leaving the batch processor at its default size on the metrics pipeline.
• Sampling in the SDK and relying on Transaction Search indexing rules, halving visibility twice over.
• Putting request IDs, user IDs, or raw paths into metric attributes.
• Running a gateway pool with no alerting on the collector’s own health, so saturation is invisible until you need it.

Best practices

• Keep instrumentation vendor-neutral. Plain OpenTelemetry SDK APIs, with the collector owning every AWS-specific decision.
• Set service.name, service.version, and deployment.environment as resource attributes everywhere. Almost every correlation feature depends on them.
• Alert on exporter failure counters and queue depth, and route those alerts somewhere that doesn’t depend on the pipeline being healthy.
• Version the collector config in Git and deploy it like application code, with a staging pipeline you can break safely.
• Redact in a processor before export. Once telemetry reaches a backend, removing it is a support ticket.
• Test a second exporter early, even to a throwaway account. A vendor migration is the wrong time to learn the fan-out path doesn’t work.

Frequently asked questions

Do I still need a collector if CloudWatch accepts OTLP directly?

Not to get data in, no. You need one to control what happens before it leaves: batching to fit endpoint limits, tail sampling, redaction, and fan-out. Small serverless estates can reasonably skip it. Anything with a dozen services will want one.

Can I send OTLP to CloudWatch over gRPC?

No. The CloudWatch OTLP endpoints are HTTP 1.1 only, accept binary or JSON payloads, and support gzip or no compression. Your collector can still receive gRPC from applications, it just can’t forward over it.

What’s the difference between ADOT and the upstream collector?

Same codebase, different build. ADOT is AWS’s tested distribution with AWS components bundled and AWS Support behind it; upstream Contrib carries a wider component set and moves faster. Configuration is compatible either way, so it’s a support and packaging choice, not an architectural one.

Do I have to enable Transaction Search to send traces?

For the X-Ray OTLP endpoint, yes. It’s a prerequisite, and enabling it routes span ingestion through CloudWatch Logs. Using the awsxray exporter against the classic X-Ray API instead doesn’t require it, but you give up the span-level analytics Transaction Search provides.

Can I ship telemetry to CloudWatch from outside AWS?

For metrics and logs, yes, using bearer token authentication instead of SigV4, which removes the need for AWS credentials on the host. Traces still require SigV4. Never hardcode the token in the config; read it from a mounted secret file or an injected environment variable.

Is it urgent to migrate off the X-Ray SDK?

Not an emergency, but it’s on a clock: maintenance mode now, end of support 25 February 2027. Existing applications keep working, they just won’t get new library instrumentation. Plan the migration on your schedule rather than someone else’s.

The one thing worth remembering

A pipeline that returns 200 is not a pipeline that works. Judge every design decision in OpenTelemetry on AWS against one question: when this drops data, do I find out from a dashboard or from a colleague asking why a service vanished off the map?

Size batches against the published limits, keep unbounded identifiers off metric attributes, decide temporality in one place, and instrument with plain OpenTelemetry so the export target stays a config line. Do that and “instrument once, export anywhere” stops being a slogan and becomes a property you can test.

Need help with your OpenTelemetry pipeline on AWS?

Most of this work is unglamorous and specific, which is why it gets deferred. Things I can help with:

• Auditing an ADOT or upstream collector config for silent drops, oversized batches, and limit breaches
• Designing collector topology for ECS, EKS, or Lambda, including gateway sizing and autoscaling
• Migrating X-Ray SDK and daemon workloads to OpenTelemetry ahead of end of support
• Cutting telemetry spend through sampling policy, cardinality control, and attribute pruning
• Dual export to CloudWatch and a third-party backend, so a future switch is a config change
• Meta-monitoring that tells you the pipeline is broken before your users do

Send me a collector config, an exporter log, or a screenshot of the gap in your dashboard and I’ll tell you what I’d look at first.

The post Instrument Once, Export Anywhere: OpenTelemetry on AWS With ADOT appeared first on John Nessime.

Two source records had been collapsed into one. Flat 12 and Flat 12A in the same block: same postcode, same rooftop geocode, both two-bed, both published the same week by different agencies. One record won the price field. The other won the contact block. What came out the far end was a listing for a property that does not exist.

That is what makes property listing deduplication hard, and it is not the problem people plan for. This post covers candidate generation, match scoring, survivorship rules, the time axis almost nobody models, and how each fails in production.

Under-merging is visible. Over-merging is not.

Get this asymmetry wrong and every threshold downstream is mistuned.

Fail to merge two records for the same property and a user sees two cards for one house. It looks sloppy, someone reports it, you fix it that afternoon.

Merge two different properties and nothing looks broken. The card renders. Real price, real photos, real agent, all belonging to different homes. Filters return it for the wrong bedroom count. Price analytics inherit a value never asked for that unit. A lead goes to an agency that never had the instruction. Nobody reports it, because nobody can see it.

A missed merge is a duplicate. A false merge is corrupted data with a plausible face on it.

Why property listing deduplication is not a string-matching problem

The obvious first attempt is to normalise the address, hash it, and group by the hash. It gets you further than you expect and then stops dead, because portals do not disagree on addresses randomly. They disagree structurally.

• One portal puts the sub-unit in its own field, one puts it in line one, one drops it.
• New-build stock is marketed by plot number long before it is marketed by street address.
• Rural properties are named rather than numbered, and the name is what agencies edit most.
• Some agencies fuzz the address on tenanted stock and publish only a street or postcode sector.

Normalisation still earns its place, it just is not the decision. libpostal is the standard tool: a C library trained on open address data, with bindings for Python, Go, Ruby, Java and Node, plus a PostgreSQL extension.

-- Via the pgsql-postal extension, which wraps libpostal.
-- postal_normalize returns every expansion that "makes sense",
-- so you index all of them rather than betting on one.
SELECT unnest(postal_normalize('Flat 12a, 12 Bramble Ct., Manchester'));

Expand rather than collapse. You do not know which spelling the other portal chose, and matching on set intersection is more forgiving than picking a canonical form and hoping both sides picked the same one.

Look for a real identifier first

Probabilistic matching is what you do when there is no key. Sometimes there is one, and it is worth a lot of scoring code.

• Great Britain: the UPRN. Assigned to every addressable location by local authorities and Ordnance Survey, it survives renaming, renumbering and demolition. OS Open UPRN publishes identifiers and coordinates as open data. Same UPRN on both sides ends the argument.
• United States: inside a single MLS, the RESO Data Dictionary gives a stable listing key and the RESO Web API is the current transport. RETS is deprecated and boards have been switching it off. The key is stable within an MLS, not across them, so overlapping markets put you back into probabilistic matching.
• Open cadastre: a parcel or title identifier is the best anchor available, with the caveat that one parcel can hold many dwellings.

Resolve on the identifier first and let the fuzzy matcher handle the remainder. That set is smaller and stranger than you expect, which is fine, because the strange ones are where false merges live.

Blocking: never compare every record with every record

All-pairs comparison is quadratic. A million listings is roughly half a trillion pairs. You do not need a bigger machine, you need to stop generating pairs that were never going to match. Blocking produces candidates cheaply from an index, and only candidates get scored.

A good key is selective enough to shrink the space and stable enough that both portals agree on it. Those pull against each other. Postcode plus bedroom count is selective, but bedroom count is one of the most disputed fields in any feed, because portals count studies, box rooms and loft conversions differently. Block on it and you discard true matches before scoring runs.

Keys that hold up tend to be geometric or structural rather than descriptive:

• A geohash or H3 cell around the geocoded point, plus its neighbours, so pairs straddling a cell edge survive.
• Postcode or postal sector combined with the numeric part of the street address.
• A trigram index on the normalised street name, which tolerates abbreviations that break exact keys.
• The agency contact number, which catches one agency’s stock syndicated under wildly different formatting.
• A perceptual hash of the lead photo, which catches what the address fields lost.

CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;

CREATE INDEX ON listing USING gist (geog);
CREATE INDEX ON listing USING gin (street_norm gin_trgm_ops);

-- The % operator compares against pg_trgm.similarity_threshold,
-- which is a session setting, not a hardcoded constant.
SET pg_trgm.similarity_threshold = 0.6;

-- Candidate pairs only. b.id > a.id yields each pair once.
SELECT a.id AS left_id, b.id AS right_id
FROM listing a
JOIN listing b
  ON b.id > a.id
 AND ST_DWithin(a.geog, b.geog, 75)
 AND a.street_norm % b.street_norm
WHERE a.source_portal <> b.source_portal;

ST_DWithin on a geography column takes metres and uses the GiST index. Seventy-five is deliberately generous: rooftop and street-interpolated geocodes for one address routinely land tens of metres apart. Strictness belongs in the score, not the block.

Every key has a blind spot, so run three or four narrow passes and union the results. That is far cheaper than one pass loose enough to cover everything. Track what each pass contributes alone, because passes rot: a feed changes, a pass stops producing candidates, and recall drops without a single error in the logs.

Scoring: what each field is actually worth

The instinct is to weight fields by how much you trust them. That is the wrong axis. The right one is how surprising the agreement is.

This is the Fellegi-Sunter model, the standard framing for record linkage. Two probabilities matter per comparison: how likely the fields agree given the records are the same property, and how likely they agree given they are not. The ratio is the evidence. Multiply across fields for a match score.

Cardinality decides the weight

Two listings agreeing on “detached house” tells you almost nothing, because a large share of the market is detached houses. Agreeing on an unusual property name in a small postcode sector tells you a great deal, because there are very few ways for that to happen by accident.

So a field’s weight is not constant. Agreeing on a postcode in a dense city centre is weak evidence; the same agreement in a sparse rural area is strong. Term-frequency adjustment handles that per value rather than per column, which matters in property data because property data is geographically lumpy.

Splink is the one I reach for first. It implements Fellegi-Sunter with expectation-maximisation parameter estimation, runs on DuckDB locally and Spark for larger jobs, supports term-frequency adjustments, and shows which field contributed what to a score. That last part matters: “the model said so” is not an answer you can act on. The trade-off is that an unsupervised model needs no labelled data but is only as sane as your blocking.

Photos are strong evidence until they aren’t

Two portals carrying one property usually carry the same photo set from the same agency, occasionally recompressed. A perceptual hash survives that, because it fingerprints visual structure rather than bytes.

from PIL import Image
import imagehash

# hash_size=16 gives a 256-bit hash instead of the 64-bit default.
# More bits means fewer accidental collisions between two beige
# living rooms shot from the same corner with the same wide lens.
a = imagehash.phash(Image.open("portal_a_01.jpg"), hash_size=16)
b = imagehash.phash(Image.open("portal_b_03.jpg"), hash_size=16)

distance = a - b   # subtraction returns the Hamming distance
print(distance)

Use the count of near-identical pairs across both sets, not the best single pair. One shared photo can be a stock exterior. Six is not.

Now the trap that produces false merges at scale. New-build schemes reuse the same CGI renders, show-home interiors and drone shots across every plot. Photo evidence will insist that forty distinct apartments are one apartment, and it will insist confidently, because the agreement really is surprising.

Treat photo agreement as conditional. Count how many distinct listings share each hash across the corpus, then downweight or discard hashes appearing on more than a handful. Same term-frequency logic, applied to images. Watermarks are the other tripwire: a portal stamping its logo into the corner shifts the hash for every photo it serves.

Three outcomes, not two

A score comes out. What you do with it should not be a single threshold.

1. Below the lower threshold: treat as distinct. Two cards.
2. Between the thresholds: hold for review. Do not merge, do not discard, surface the pair with its field-level evidence.
3. Above the upper threshold: merge automatically.

The width of that middle band is a budget decision, not an accuracy decision. Widen it and you catch more ambiguous pairs before they do damage, at the cost of review time. Narrow it and you are choosing which error you will eat. Because over-merging is the expensive one, the band should sit asymmetrically.

One rule with no downside: clustered records keep their source rows. Store the cluster assignment in a separate table keyed by source record ID, never by overwriting the source. When a merge turns out wrong, you want an UPDATE on a mapping table rather than a restore from backup.

Survivorship: which record wins each field

Merging decides records belong together. Survivorship decides what the merged record says, and it is where the opening bug came from. Do not pick a winning record. Pick a winning source per field, and write it down.

• Price: most recently observed, tie-broken by source rank. Staleness is the main risk.
• Agent and contact: highest-ranked source only, never blended. A number and an agency name are one unit; splitting them produces a contact that does not exist.
• Floor area and bedrooms: the source with the most complete structured fields, and flag the cluster when sources disagree. Disagreement here often means the merge itself is wrong.
• Photos: union, deduplicated by hash, ordered by highest-ranked source.

-- Winning price per cluster: freshest observation,
-- with source_rank breaking ties on identical timestamps.
SELECT DISTINCT ON (cluster_id)
       cluster_id,
       price,
       source_portal,
       observed_at
FROM listing_price_observation
ORDER BY cluster_id,
         observed_at DESC,
         source_rank ASC;

Keep provenance on the output. Every surviving field should carry which source it came from and when it was observed. That is the difference between debugging a bad merge in ten minutes and debugging it in two days.

The time axis nobody models

A property is listed, withdrawn, then relisted months later with a different agency, a different price and new photos. One entity or two?

Both. There are two entities and most pipelines model only one:

• The property: physical and permanent. Same building, same UPRN or parcel, forever.
• The listing event: commercial and episodic. One campaign, one agency, one price band.

Collapse them and you get a record whose price history is a jagged line with no explanation, whose days-on-market figure is meaningless, and where a price cut and a relisting are indistinguishable. Separate them and both questions get easy: duplicates resolve at the property level, campaign analytics at the event level.

My rule for splitting events: a gap in observation beyond some window, combined with a change of agency or a price move outside a tolerance, starts a new event on the same property. Tune the window per market.

Troubleshooting: where these pipelines break

Cluster count collapses overnight

Almost always a blocking key that became non-selective. A portal starts sending a placeholder coordinate for missing geocodes, ten thousand listings land on one point, and that becomes a single enormous block. Alert on largest-block size, not just job duration. A pass that generates one huge block runs slowly first and merges wrongly second.

One cluster swallows a whole street

Transitive closure doing what it was told. A matches B, B matches C, so all three are one cluster even though A and C are plainly different. Cap cluster size and quarantine anything over it. Check internal consistency too: a cluster with three bedroom counts and a fifty percent price spread is wrong regardless of the pairwise scores.

Fresh duplicates appear every single day

The source record IDs are not stable. A portal that regenerates its internal ID on every republish produces a new record daily, and a cluster mapping keyed on that ID re-resolves from scratch every run. Key the mapping on a fingerprint you control, built from fields that do not churn.

The run got slow and nothing changed

Check the trigram and GiST indexes are still being used. A change in normalisation that alters the distribution of street_norm can push the planner onto a sequential scan, and the join goes quadratic with no error anywhere. Run EXPLAIN ANALYZE on the candidate query before adding hardware. When you do need hardware, entity resolution is memory-hungry rather than CPU-hungry and spikes during pair scoring, so a single large-RAM instance from a provider like Contabo or InterServer is often cheaper than a managed cluster you need for one stage of one job.

Common mistakes

• Tuning to a single accuracy number. Precision and recall are not interchangeable here.
• Destroying source records on merge. Every merge you cannot reverse is a merge you cannot fix.
• Blocking on a disputed field. Bedroom count and property type feel selective and are exactly what portals disagree on.
• Trusting geocode proximity alone. Two flats in one block are metres apart and are not the same property.
• Dropping the sub-unit. Discard “Flat 2” because it did not parse and you have built a machine that merges apartment buildings into single homes.
• No feedback loop. Review decisions are labelled training data arriving for free.

Best practices

• Hold out a hand-labelled sample of a few hundred pairs, weighted toward hard cases, and score every model change against it.
• Make every merge explainable at field level. If you cannot show why, you cannot defend it to the person whose listing you broke.
• Keep the review queue small enough that it is actually worked. An unread queue is auto-merging with extra steps.
• Stamp each cluster with the rule version that produced it, so changes can be evaluated rather than guessed at.
• Emit merge rate, split rate, queue depth, largest block size and per-pass contribution as first-class metrics. A managed backend such as Grafana Cloud saves running another stateful service, though self-hosted Prometheus is fine if you already have it.
• Cache photo hashes in object storage keyed by source image URL. Hashing is the slow part of every rerun and never needs doing twice. Storage without egress fees, such as Cloudflare R2, fits when hashing and storage are not colocated.

FAQ

Can I just match on address and postcode?

For one market with clean feeds and no apartment blocks, further than you would think. It breaks on sub-units, named properties, plot-marketed new-builds and portals that abbreviate differently. Treat exact address matching as one high-confidence signal, not the whole decision.

Which library should I start with?

Splink for the model, libpostal for address normalisation, your existing database for blocking. Splink’s advantage is that it explains its scores. Zingg and the Python Record Linkage Toolkit solve adjacent problems and suit teams that prefer active learning or lightweight prototyping.

Do I need machine learning for property listing deduplication?

Not to start. Deterministic rules on a good identifier plus a handful of weighted comparisons handle most pairs. The probabilistic model earns its place on the residue, which is where the expensive mistakes live. Build the deterministic layer first so you can measure what is left.

How do I handle deliberately hidden addresses?

Lean on non-address signals: photo hashes, floor area, agency identity, price. Accept that some pairs are unresolvable and design the queue to hold them rather than forcing a decision. A permanently unresolved pair beats a confidently wrong merge.

Should one property from two agencies be one record or two?

A product decision, not a technical one, and worth settling before you build. Multi-agency instructions are legitimate. Resolving to one property with several current listing events keeps both facts intact and leaves the display choice to the front end.

How often should the pipeline re-resolve?

Incrementally on every ingest, so new listings join existing clusters immediately, plus a full rebuild on a slower cadence to pick up rule changes and correct drift. The rebuild is the step people skip, and it catches clusters that were correct when formed and are not any more.

The one thing worth remembering

Property listing deduplication is not really a matching problem. It is a problem of asymmetric error costs wearing a matching problem’s clothes.

A duplicate card is visible, cheap and gets reported. A false merge is invisible, plausible, and quietly poisons everything downstream. Build your thresholds, blocking passes and survivorship rules around that asymmetry, keep the source records so mistakes stay reversible, and put a human in the band where the evidence is not conclusive. The rest is tuning.

Need help with a multi-portal listing pipeline?

This is the kind of work I do. If you are running property data from several portals and something is not adding up, I can help with:

• Designing blocking and match scoring for cross-portal feeds, including the multi-pass strategy and the metrics to watch it with.
• Auditing an existing dedupe pipeline for false merges, oversized clusters and passes that quietly stopped contributing.
• Building the two-level property and listing-event model so price history, days on market and duplicate suppression stop fighting each other.
• Setting up survivorship rules with field-level provenance, so a bad merge is a ten-minute fix rather than a restore.
• Migrating ingestion off deprecated transports onto current portal or MLS APIs without losing history.
• Right-sizing the infrastructure so the resolution job runs on something appropriate rather than whatever was already there.

Send me two records that should have matched and didn’t, or two that shouldn’t have and did, and I’ll tell you what’s going on.

The post One Home, Six Portals: Property Listing Deduplication Without False Merges appeared first on John Nessime.

That is the failure mode worth understanding, and it stays invisible until you go looking for it. The conversion was real. The files did get much smaller. But the dashboard queries were still selecting every column from an unpartitioned table, so the engine still had to open every column chunk in every file. Columnar storage only pays out when the query gives the reader something to skip.

This is a working comparison of Parquet vs JSON vs CSV on the two things that end up on an invoice: bytes read, and time spent reading them. Each format gets a profile with the cases where it wins and the cases where it does not, followed by the places where the promised savings quietly evaporate and a procedure for modelling the change on your own data before you spend an engineering week on it.

Your billing model decides how much the format matters

Before comparing formats, work out what you are actually charged for. The same conversion produces very different savings depending on which of these you are on.

• Bytes scanned from object storage. Amazon Athena is the clearest case. You are billed for the bytes a query reads out of S3, rounded up, with a minimum charge per query. Here a format change converts almost directly into money.
• Logical bytes processed. BigQuery’s on-demand model bills on the uncompressed logical size of the columns a query references in a native table, not on the compressed size sitting on disk. Compressing harder does not lower that bill. Touching fewer columns does.
• Compute time. Snowflake, Databricks, EMR, and anything you run on your own hardware charge you for how long a machine is awake. The format shows up as wall clock time, which is a real cost but a much softer one.

The BigQuery detail catches people out. Google documents that on-demand cost is calculated from logical, uncompressed bytes for native tables, but that when you query external data stored as Parquet or ORC, the bytes charged are limited to the columns BigQuery actually reads. The same file format therefore sits on opposite sides of the billing line depending on whether the data was loaded or is being referenced in place. Both Athena and BigQuery also apply a 10 MB minimum per query, and BigQuery applies it per table referenced, so a swarm of small queries against small tables costs far more than the data volume suggests.

CSV: cheap to produce, expensive to interrogate

CSV has no schema, no statistics, and no internal structure. Every value is text. The number 1000000000 takes ten bytes instead of the four it would take as a 32-bit integer, and because the layout is row-wise, there is nothing for a column-aware compressor to exploit.

Where CSV wins

• A human needs to open it. Nothing else comes close.
• Export from a legacy system that offers no other option.
• Small data. Below a few megabytes, the metadata and encoding overhead of a columnar file is not repaid.
• Streaming appends. You can write a line at a time and stop whenever you like.

Where CSV does not

Two problems, and the second is worse than the first.

The first is that the reader has to parse every byte, even the bytes it intends to throw away. The DuckDB team published a comparison on TPC-H data at scale factor 20, run on a laptop, reporting the median of five runs, and it is the most honest set of numbers I have seen on this. Loading the lineitem table took roughly 11.8 seconds from CSV against 5.2 seconds from Snappy-compressed Parquet, with the CSV file about five times larger on disk. That is a 2x gap on load, not the order of magnitude most people assume. Modern CSV readers are genuinely fast.

The gap opens when you query the files directly rather than loading them. On the same setup, TPC-H Q1 ran in about 6.7 seconds against CSV and 0.9 seconds against Parquet, and the join-heavy Q21 took roughly 20 seconds against 2.1. That difference is not parsing throughput. It is that the Parquet reader can skip row groups using statistics and skip unreferenced columns entirely, while the CSV reader cannot skip anything, because nothing in the file tells it what it would be skipping.

The second problem is dialect. There is an RFC for CSV and it is widely ignored. Quoting, escaping, embedded newlines, date formats, and whether NA means null or Namibia are conventions rather than rules, so every consumer re-guesses and sooner or later one guesses differently from the others. That failure does not raise an error. It produces wrong numbers in a report nobody questions.

JSON: you pay for the schema on every single row

Newline-delimited JSON is what most event pipelines emit, because a producer can add a field without coordinating with anyone downstream. That flexibility is worth something real. It is also why JSON is usually the most expensive format on a bytes-scanned bill: every record repeats every key name, so a field called customer_subscription_status carries those thirty-odd bytes on every row, forever.

Where JSON wins

• Landing zone for data whose shape you do not control. Accept it, keep the raw copy, argue about the schema later.
• Genuinely nested or sparse records, where flattening into columns would give you a table that is mostly nulls.
• Debugging. Being able to decompress one file and read a record with your eyes is worth a lot at three in the morning.
• API and webhook payloads, where it is simply what arrives.

Where JSON does not

As a query target at any serious volume. You are paying scan charges on repeated key names, on quote characters, and on the fact that every number is stored as text. Compressing the files helps storage and helps the scan bill on Athena, but it cannot give the engine anything to skip, and gzip brings a separate problem covered below.

The nested-data argument for keeping JSON has also weakened. Parquet now has a ratified Variant type for semi-structured data, with support arriving across Delta Lake, Iceberg, and Spark, plus a shredding scheme that pulls frequently accessed fields out into real columns while leaving the rest flexible. If you are keeping JSON purely because your records have unpredictable shape, that reason is worth revisiting rather than treating as settled.

Parquet: what the file is actually doing

Understanding the layout is what lets you predict whether a migration will pay for itself, so it is worth thirty seconds.

A Parquet file is divided into row groups. Inside each row group, every column is stored as its own contiguous column chunk, and each chunk is split into pages. The footer at the end of the file carries the schema and, per column chunk, statistics such as minimum, maximum, and null count. That structure is what makes the two headline optimisations possible:

• Column pruning. The reader consults the footer, works out the byte ranges of the three columns you asked for, and never issues a read for the other forty-seven.
• Predicate pushdown. If a row group’s maximum value for event_date falls below your filter’s lower bound, the entire row group is skipped without being read.

Two later additions refine this. The page index stores per-page minimum and maximum values in a structure near the footer, so a reader doing a selective scan can locate matching pages without walking every page header, which previously meant pulling most of the column off disk anyway. Bloom filters cover the case statistics cannot help with at all: a high-cardinality column such as a user ID, where every row group’s min and max span the whole value range and pruning by range is useless. A bloom filter can say that a value is definitely not present in a row group. DuckDB even exposes parquet_bloom_probe so you can check which row groups a given value would eliminate.

Encoding does the rest of the work. Because a column chunk holds values of a single type, dictionary and run-length encoding apply before any general-purpose compressor starts. Low-cardinality columns such as country codes or status flags compress extremely well. UUIDs and free text do not, and writers often disable dictionary encoding for them. This is why compression ratios quoted in blog posts are close to meaningless for your data: the ratio is a property of your cardinality, not of the format.

Where Parquet does not win

• Point lookups. Fetching one complete row by ID means touching every column chunk. A row format reads a single contiguous block instead.
• Row-level updates. Parquet files are immutable. Changing one row means rewriting the file, which is precisely the problem Iceberg and Delta Lake exist to solve on top of it.
• Streaming writes. The schema lives in the footer, so a writer has to finish the file before it is readable. Small-batch writers produce small files, which carries its own tax.
• Tiny datasets. Below a few megabytes, footer and row group overhead is not worth paying.
• Human inspection. It is binary. You need tooling, and someone will eventually ask you for a CSV export anyway.

Where the savings quietly disappear

Every one of these has cost someone a migration that looked excellent in the design document.

• Small files. Each file costs a metadata read, an object storage request, and reader setup. Thousands of 2 MB Parquet files will lose to a handful of large CSVs. The commonly cited target is roughly 128 MB to 1 GB per file, and you need a compaction step to get there if your writer emits micro-batches.
• One enormous row group. Parallelism in most engines is per row group. A file containing a single row group is processed by a single thread no matter how many cores you have. Check this rather than assuming the writer got it right.
• Select-everything queries in the reporting layer. Column pruning cannot help a query that asks for all columns. This is the one from the opening paragraph, and it is usually a BI tool doing it rather than a person.
• No partitioning. Format reduces bytes per row. Partitioning reduces how many rows are considered at all. The two multiply, and partitioning is normally the larger lever. Converting without partitioning leaves most of the money on the table.
• Gzipped text. AWS documents that Parquet and ORC are always splittable, because they compress sections independently and carry metadata pointing at those sections, while most text compression formats are not. Bzip2 and LZO are splittable; gzip is not. One 5 GB gzipped CSV cannot be divided among workers, so it is read start to finish by one reader. The compression made storage cheaper and the query slower.
• Uppercase file extensions. A small operational trap worth knowing: Athena determines the compression type of CSV and JSON data from the file extension, does not recognise uppercase extensions such as .GZ, and treats a file with no extension as uncompressed plain text.

How to model the change before you migrate

Do not migrate on faith. This takes an afternoon and answers the question properly.

1. Find out what you scan today. Every Athena query execution carries a DataScannedInBytes statistic. Pull it for a representative window and rank queries by bytes scanned. Almost always a handful of them are most of the bill, and those are the only ones worth optimising.
2. Convert one day of data, not all of it. A single partition is enough to measure with.
3. Look inside the resulting file before trusting it. Row group count, row group size, and compression codec decide whether the file behaves.
4. Re-run the same queries against the converted partition and compare bytes scanned. Measured, not estimated.
5. Price the write side. Conversion is not free. Glue jobs, EMR time, and Lambda invocations all cost money, and if a dataset is queried twice a month the conversion may never pay back.
6. Check who else reads the files. One downstream consumer that only speaks CSV turns a clean migration into a dual-write pipeline.

For step three, DuckDB is the fastest way to inspect a file without standing up a cluster. It runs happily on a laptop or a small VPS from a provider like Contabo or InterServer, which is usually all the compute this kind of analysis needs:

-- one row per column chunk: row groups, sizes, codec
SELECT row_group_id, row_group_num_rows, row_group_bytes,
       path_in_schema, compression
FROM parquet_metadata('events.parquet')
LIMIT 20;

-- convert a sample and control the row group size
COPY (SELECT * FROM 'events.json')
TO 'events.parquet'
(FORMAT parquet, COMPRESSION zstd, ROW_GROUP_SIZE 1000000);

The parquet_metadata function reads the footer and returns a row per column chunk, so you can see straight away whether your writer produced one row group or two hundred, and which codec it used. If you would rather stay in Python, pyarrow.parquet.ParquetFile exposes the same footer through its metadata attribute, including per-column statistics and whether a bloom filter offset is present.

If the data already lives in S3 and you want the engine you are already paying for to do the conversion, Athena’s CTAS handles it in one statement. The AWS documentation lists GZIP and SNAPPY as the compression options for Parquet output here, with GZIP as the default:

CREATE TABLE events_parquet
WITH (
  format = 'PARQUET',
  write_compression = 'SNAPPY',
  external_location = 's3://your-bucket/events-parquet/',
  partitioned_by = ARRAY['event_date']
) AS
SELECT event_id, user_id, event_type, amount, event_date
FROM events_json;

Three things to know before running it. Partition columns have to come last in the SELECT list. CTAS refuses to write into a location that already contains data, so a re-run needs the prefix cleared first. And Athena has a write limit of 100 partitions per CTAS statement, which means a large backfill has to be chunked rather than fired off in one go.

How I would decide

• Keep the raw landing zone in whatever arrives. Usually JSON. Do not convert on ingest and discard the original; you will want it the first time a schema assumption turns out to be wrong.
• Convert once, at the point where data becomes queryable. Partitioned Parquet for anything a dashboard or an analyst touches repeatedly.
• Partition before you optimise the format. If there is budget for exactly one change, partition on the column your queries actually filter on.
• Choose the codec by read frequency. Zstd where scanned bytes are the bill and the data is read often, Snappy where decompression speed matters more, gzip only when a consumer demands it.
• Leave small CSVs alone. Reference tables, seed data, config. Converting a 4 MB lookup file to Parquet is busywork.
• Watch the scan bill, not the storage bill. Storage is cents. Scans are dollars. Athena workgroups can cap bytes scanned per query, and that limit is worth setting before someone discovers what a full table scan costs.

Frequently asked questions

Is Parquet always cheaper than CSV?

On a bytes-scanned billing model, for analytical queries that touch a subset of columns, almost always. It is not cheaper for point lookups, for very small datasets, for row-level updates, or when the query selects every column and filters on nothing. It also costs compute to produce, which matters when the data is rarely read.

How much smaller is Parquet than CSV in practice?

It depends entirely on cardinality, and anyone quoting a single ratio is guessing about your data. In the DuckDB TPC-H comparison the Parquet file came out around five times smaller than the CSV. Columns with few distinct values do far better than that; UUID and free-text columns do far worse. Measure on one partition of your own data instead of trusting a headline number.

Does compressing my JSON or CSV give me the same savings?

Partially. On Athena you are billed for bytes scanned before decompression, so compression does reduce the bill. What it cannot do is let the engine skip columns or row groups, and if you reach for gzip you lose splittability, which can make queries slower even as they get cheaper. Compression and columnar layout solve different halves of the problem.

Should I use Parquet or ORC?

For most teams that difference is much smaller than the difference between either of them and CSV. Parquet has wider support across engines and languages, which is usually the deciding factor. ORC has a long history in Hive-centric stacks and remains a reasonable choice if that is where you already live. Both are splittable and both support predicate pushdown.

Why did my costs not drop after converting to Parquet?

The usual suspects, in the order I would check them: the table is not partitioned, the queries select all columns, the files are too small, or the writer produced one enormous row group. Pull DataScannedInBytes for the same query before and after. If the number barely moved, the engine was not able to skip anything, and the problem is the query or the layout rather than the format.

Is JSON still worth keeping now that Parquet has a Variant type?

As a raw landing format and a transport format, yes. As the thing analysts query directly, the case is weaker than it used to be. Variant support is still spreading across engines, so confirm your query engine handles it before designing around it rather than assuming it is available everywhere.

What file size should I aim for?

Roughly 128 MB to 1 GB per file is the widely used guidance, with row groups sized so a file holds several of them rather than one. Below that range you pay per-file overhead on every query; above it you can starve parallelism if the row group layout is wrong.

The one thing worth remembering

The Parquet vs JSON vs CSV question is not really about the formats. It is about how much of your data the engine is permitted to ignore. Parquet hands an engine the most opportunities to skip work, JSON gives it almost none, and CSV sits between the two while making every reader re-guess the schema.

An opportunity to skip work is not the same as skipping it, though. If the query asks for every column, if the table is not partitioned, if the files are too small or the row groups too large, you have paid the conversion cost and bought a smaller file with the same bill attached. Measure bytes scanned before and after on a single partition. That one number tells you the truth in an afternoon, and it beats any comparison table you will read, including this one.

Need a second pair of eyes on your data lake costs?

Most format work starts as a cost problem and ends as a layout problem. Things I can help with:

• Auditing which queries actually drive your Athena or BigQuery bill, and whether a format change would move them at all
• Designing the partition scheme and file sizing before the conversion, so the migration is worth running
• Building the CSV or JSON to partitioned Parquet conversion as a Glue, Spark, or DuckDB job, including compaction for small-file pipelines
• Diagnosing conversions that did not pay off: row group layout, dictionary encoding, splittability, gzip traps
• Setting scan limits, workgroup controls, and alerting so one bad query cannot produce a surprise invoice
• Retrofitting Iceberg or Delta Lake where the real requirement is row-level updates rather than a different file format

If you want a concrete opinion rather than a general one, send me a query’s execution statistics along with the table schema and a listing of one partition, and I will tell you where your bytes are going.

The post Parquet vs JSON vs CSV: Where the Money Actually Goes appeared first on John Nessime.

That asymmetry, sending works and receiving doesn’t, is the single most useful clue in the whole problem. It narrows the search enormously. Yet most guides on Zoho Mail not receiving email open by telling you to check your SPF record, which has nothing whatsoever to do with inbound delivery. People lose a full day tuning an SPF string while their MX records quietly point at a data center that has never heard of their domain.

This guide walks the inbound path in the order a sending mail server actually walks it, groups the real causes by failure family, and gives you the exact commands to prove which one you’re looking at. Everything here applies whether your DNS lives at Cloudflare, Namecheap, GoDaddy, Route 53, or a cPanel box at InterServer or Contabo.

Sending and receiving are two separate systems

This is worth internalizing before you touch a single DNS record, because it eliminates most of the internet’s advice on this topic in one move.

When you send from Zoho, your client authenticates against Zoho’s outbound SMTP server with a username and password. Your domain’s DNS plays no part in that handshake. SPF, DKIM and DMARC records exist so that the recipient’s server can decide whether to trust mail claiming to be from you. They are judgements other people make about your outbound mail.

When someone sends to you, none of that matters. Their server looks up your MX records, connects to whatever hostname it finds, and hands the message over. Your SPF record is never consulted. Your DKIM key is never consulted.

So if outbound works and inbound doesn’t, the fault is in the MX lookup, the connection that follows it, or Zoho’s decision about what to do with the message once it arrives. Not SPF. There is one narrow exception, covered later, where Zoho’s own inbound spam checks reject mail because the sender’s SPF fails. That is a different record on a different domain.

The inbound path, in the order it actually happens

Six things happen between someone hitting send and a message appearing in your Zoho inbox:

1. The sending server splits the recipient address and takes the domain part.
2. It asks DNS for the MX records of that domain, following delegation from the root down to whichever nameservers the registry says are authoritative for you.
3. It sorts the results by preference number and takes the lowest one first.
4. It resolves that hostname to an address record.
5. It opens an SMTP connection and offers the message with MAIL FROM and RCPT TO.
6. The receiving server decides whether that recipient exists in its organization and where to file the message.

Steps two through four are pure DNS. Step five is network. Step six is Zoho configuration. Diagnosing this well is mostly a matter of finding out which step you’re stuck on instead of guessing.

Five DNS failure families behind Zoho Mail not receiving email

Nearly every inbound failure I’ve seen falls into one of these. They present almost identically from the user’s chair, which is why guessing goes badly.

1. You’re editing records at a provider that isn’t authoritative

This is the most common one and the most frustrating, because the records look perfect in the control panel you’re staring at. They’re just not the records the internet reads.

It happens whenever the domain is registered in one place and the nameservers point somewhere else. You buy at Namecheap, delegate to Cloudflare, then add MX records back in the Namecheap DNS panel out of habit. Or a previous developer moved DNS to a hosting account you no longer log into. The registrar panel accepts your edits and stores them faithfully. Nobody queries them.

Find out where authority actually sits:

# Which nameservers does the parent zone delegate to?
dig +short NS example.com

# What do those nameservers say about your MX, directly?
dig MX example.com @ns1.yourprovider.com +norecurse

# Watch the full delegation chain from the root
dig +trace MX example.com

The +trace output is the one to read carefully. It shows each referral step, so you can see exactly which nameserver set the world is being pointed at. If those names don’t match the provider where you’ve been editing, you’ve found your problem and the fix takes thirty seconds.

2. Correct-looking MX records for the wrong Zoho data center

This is the failure mode that stays invisible longest, and it’s the reason copying MX values off a random blog post is dangerous.

Zoho runs regional data centers, and your account lives in exactly one of them. The mail hostnames differ by region: the international service uses .com hostnames, and the European and Indian deployments use their own regional equivalents. Your domain’s mailboxes exist only in the data center where your organization was created.

Point your MX at the wrong region and everything looks healthy. The records resolve. The hostnames are genuine Zoho servers. A lookup tool will show green. But mail arrives at infrastructure that has no record of your domain, and senders get a rejection saying the address doesn’t exist, or nothing at all.

For the international service, Zoho’s documentation specifies three records, in this order of preference:

Priority 10   mx.zoho.com
Priority 20   mx2.zoho.com
Priority 50   mx3.zoho.com

Do not take that as gospel for your account. The authoritative source is the DNS Mapping section of your own Zoho Mail Admin Console, which shows the exact hostnames for the data center your organization sits in. Read them there, copy them from there. If you’re unsure which region you’re on, the domain in the URL when you log into the admin console tells you.

Verify what the world sees, not what your panel shows:

dig +short MX example.com

# Confirm the target hostname itself resolves
dig +short mx.zoho.com

3. Leftover MX records from whatever came before

MX preference is not a fallback list in the sense most people assume. A sending server tries the lowest number first, and if that server accepts the message, the transaction is over. It never touches the others.

So a stale record from your old host sitting at priority 0 or 5 doesn’t compete with Zoho. It wins outright, every time. The mail is delivered successfully, to a mailbox nobody checks, on a server that may still be quietly accepting for your domain. The sender sees no error. You see nothing at all.

This is also why partial migrations bite. Two providers both accepting mail for one domain means your messages split between them based on preference and retry timing, which looks like random message loss and is nearly impossible to reason about from the inside.

The rule is simple: after cutover, the only MX records in the zone should be Zoho’s. Delete everything else, including records for subdomains you forgot about. Check the full answer rather than a truncated view:

dig MX example.com +noall +answer

4. A CNAME sitting where the MX records live

Two related problems here, and both are structural rather than typos.

The first: an MX record must point to a hostname with an address record. Pointing an MX at a name that is itself a CNAME is forbidden by the DNS specifications, and Zoho documents this explicitly. Some resolvers cope, others don’t, and the ones that don’t will fail intermittently in ways that look like a network problem.

The second: a CNAME at the zone apex. Classic DNS forbids a CNAME coexisting with any other record at the same name, and MX records at the apex are exactly that. Providers like Cloudflare work around this with CNAME flattening, which synthesizes an address record and does keep your MX records functional. Providers with a naive implementation may shadow or drop them. If someone added an apex CNAME to point the bare domain at a hosting platform or a site builder, that’s worth checking early.

dig CNAME example.com +short
dig A example.com +short

One thing that is not a problem, despite frequent claims otherwise: Cloudflare’s orange-cloud proxy setting. Proxying applies to HTTP traffic on address records. It has no effect on MX records, and toggling it grey will not fix inbound mail.

5. DNSSEC left broken after a nameserver change

Less common, but spectacular when it happens, because it takes the entire domain down rather than just mail.

If DNSSEC was enabled and the DS record at the registry no longer matches the signing keys at your current DNS provider, validating resolvers refuse to return any answer at all. They return SERVFAIL. Non-validating resolvers still work, which is why the site loads fine for you and mail from major providers, who mostly validate, vanishes.

The test is a one-liner. +cd disables validation checking:

# Fails with SERVFAIL if validation is broken
dig MX example.com

# Same query, validation bypassed
dig +cd MX example.com

If the second command returns your records and the first doesn’t, you have a DNSSEC mismatch. The fix is at your registrar: either update the DS record to match your current provider’s keys, or remove DNSSEC entirely, confirm mail flows, and re-enable it properly afterwards.

When DNS is clean and mail still doesn’t arrive

If your MX records resolve correctly to the right data center and nothing else is in the zone, DNS has done its job. The remaining causes sit on either side of it.

Your own server is intercepting the mail

This catches people running a site on cPanel or DirectAdmin, and it’s genuinely counterintuitive. Control panels have an email routing setting per domain, typically offering local, remote, or automatic delivery. If it’s set to local mail exchanger, the mail server on that box delivers messages for your domain into its own accounts and never performs an MX lookup.

The effect is narrow but confusing: external senders reach Zoho normally, while anything sent from a script or a form on that server goes to a local mailbox instead. Contact form submissions disappear while ordinary email works. Set routing to remote mail exchanger on any host where the site lives but the mailboxes don’t.

The address doesn’t exist in your Zoho organization

Zoho will only accept mail for addresses it knows about. A user account, an alias on a user, or a group. If someone hands out an address that was never created, Zoho rejects it and the sender gets an address-not-found bounce that they may not think to forward to you.

Zoho supports a catch-all address for exactly this, and it’s worth setting one during a migration so misaddressed mail lands somewhere recoverable rather than bouncing into the void.

Zoho accepted it and quarantined it

Under Security and Compliance in the Admin Console, Zoho lets you decide what happens to mail that fails SPF, DKIM, DMARC or DNSBL checks. The available actions include delivering it, quarantining it, or rejecting it outright, with or without a bounce.

Set to reject-without-bounce, a sender whose own SPF is slightly wrong gets silently dropped, and neither party learns anything. This is the narrow SPF connection I mentioned earlier: it’s the sender’s SPF being judged, not yours. Check the incoming quarantine before concluding mail never arrived, and check the organization blocked list while you’re there, since a blocked domain or TLD produces exactly the same silence.

The domain lapsed or lost verification

If a domain expires, its DNS stops resolving and every associated record becomes unreachable, which also invalidates the domain verification Zoho performed. Zoho’s documentation notes that after renewal you have to reconfigure MX records as part of reverification. Renewing the domain alone doesn’t restore mail.

A troubleshooting sequence that ends in an answer

Work these in order. Each step either clears a layer or hands you the cause, so you never end up changing three things at once and losing track of which one worked.

1. Confirm delegation. Run dig +short NS example.com and check it matches the provider you’re editing. If not, stop here and fix that first.
2. Read the MX records from an authoritative nameserver. Use dig MX example.com @ns1.yourprovider.com so you’re seeing the source of truth rather than a cached copy.
3. Compare against the Admin Console. Open the DNS Mapping page in Zoho Mail and match hostnames character for character, including the regional suffix.
4. Confirm nothing else is in the answer. Any non-Zoho MX record is a suspect, especially one with a lower preference number.
5. Test resolution from a public resolver. Try dig MX example.com @1.1.1.1 and @8.8.8.8. Differences between them mean you’re mid-propagation and waiting is the fix.
6. Rule out DNSSEC. Compare dig MX example.com against dig +cd MX example.com.
7. Prove the SMTP conversation. Send a real test and watch what the server says.
8. Check quarantine and blocked lists in the Zoho Admin Console before concluding anything was lost.

For step seven, swaks is the cleanest tool. It speaks SMTP for you and prints the full exchange, so you see the actual response code rather than guessing from a bounce message:

# Stop after RCPT TO so no message is actually delivered
swaks --to you@example.com --server mx.zoho.com --quit-after RCPT

A 250 after RCPT TO means Zoho recognises the address and would accept the mail. A 550 means it doesn’t know that recipient, which points straight at a missing mailbox or the wrong data center. A connection timeout usually isn’t Zoho at all: many residential and cloud providers block outbound port 25 by default, so run this from a machine you know has egress on 25. If you only have a laptop on a consumer connection, testing through a VPN endpoint that permits SMTP, or from a small VPS, avoids chasing a phantom.

Common mistakes

• Fixing SPF to solve an inbound problem. Your SPF record has no role in receiving mail. If you can send but not receive, it is not SPF.
• Copying MX values from a tutorial. Regional deployments have different hostnames. Use your own Admin Console every time.
• Adding Zoho’s records without removing the old ones. A lower preference number beats Zoho outright and nothing warns you.
• Trusting the DNS panel over a query. A saved record and a published record are different things. Query it.
• Cutting MX over before creating the mailboxes. Every address that doesn’t exist yet bounces, and those messages are gone.
• Blaming propagation on day three. Propagation is bounded by the TTL that was in effect before the change. If it’s been longer than that, something is genuinely wrong.
• Trailing dot confusion. Some panels want mx.zoho.com. with the dot, others append the zone themselves and turn mx.zoho.com into an unresolvable name. Always confirm with a query afterwards.

Best practices

• Drop the TTL before you migrate. Lower the MX TTL to a few minutes a day ahead of cutover, do the change, then raise it back. This turns a rollback from an overnight wait into a coffee break.
• Create every mailbox, alias and group before touching MX. Zoho’s own guidance is to configure users first and change MX last.
• Set a catch-all during the transition. It converts “we lost an email” into “it’s in the catch-all mailbox”.
• Keep DNS in one place. One authoritative provider, one panel, no ambiguity about where records live.
• Monitor the MX record itself. An uptime or synthetic monitoring service that checks DNS answers, or a scheduled dig comparison in your own pipeline, catches an accidental zone edit before a client does.
• Enable DKIM and set DMARC after inbound is confirmed working. They matter for your outbound reputation. Doing them first just adds variables while you’re debugging.
• Keep a copy of the zone. Export it before major changes. Reconstructing a zone from memory during an outage is miserable.

Frequently asked questions

Why can I send from Zoho Mail but not receive?

Because sending authenticates directly against Zoho’s outbound servers and doesn’t consult your domain’s DNS, while receiving depends entirely on your MX records. Working outbound with broken inbound is a strong signal that the problem is an MX record, a nameserver delegation issue, or a leftover record from a previous provider.

How long should MX changes take to work?

Propagation is governed by the TTL that was set on the old record, not by any fixed schedule. If the previous TTL was one hour, resolvers holding a cached copy will refresh within an hour. If it was set to a day, expect a day. Nothing you do afterwards speeds up a cache that has already been populated.

Do I need all three Zoho MX records?

Mail will flow with only the primary, but the backups exist so that senders have somewhere to deliver if the primary is unreachable. Without them, sending servers queue and retry, which delays delivery and eventually bounces. Add the full set your Admin Console lists.

Does SPF affect whether I receive email?

Your own SPF record does not. It’s evaluated by servers receiving mail from you. The one indirect connection is that Zoho evaluates the sender’s SPF on incoming mail, and if your Admin Console is set to reject or quarantine on SPF failure, legitimate mail from badly configured senders can disappear.

Why do lookup tools say my MX records are fine when mail still fails?

Lookup tools verify that records exist and resolve. They can’t know which Zoho data center your organization lives in, whether the mailbox has been created, or whether a message is sitting in quarantine. A green result rules out one layer, not all of them.

Can I run Zoho Mail alongside another mail provider on one domain?

Not by simply listing both sets of MX records. That splits mail unpredictably. Zoho does offer routing arrangements designed for gradual migration, configured on the Zoho side rather than by stacking MX records in DNS. If you need a genuine split, use the supported mechanism.

My contact form emails vanish but normal email works. Why?

Almost always the control panel email routing setting on the server hosting the site. If it’s on local mail exchanger, mail generated on that machine is delivered into a local account rather than sent out to Zoho. Switch it to remote.

Conclusion

If you take one thing from this: when Zoho Mail is not receiving email but sending works, stop looking at authentication records. Inbound mail is decided by your MX records, and those records have to be published by the nameservers that are actually authoritative for your domain, pointing at the data center your Zoho organization lives in, with nothing else in the answer competing with them.

Query, don’t assume. Read the answer from the authoritative nameserver rather than the panel, compare it against your own Admin Console instead of a tutorial, then work down through SMTP and Zoho’s own quarantine. Almost every case resolves in that order, and the ones that don’t are usually a mailbox that was never created in the first place.

Need someone to untangle your mail DNS?

I work on DNS and mail delivery problems like this regularly, usually for people who have already tried three fixes and made things harder to reason about. Things I can help with:

• Diagnosing inbound mail failures end to end, from delegation through SMTP response codes to Zoho’s quarantine
• Planning and executing a mail migration to Zoho with a TTL schedule and a rollback path that actually works
• Auditing and cleaning up a zone that several people have edited over the years
• Setting up SPF, DKIM and DMARC properly once inbound is stable, including staged DMARC enforcement
• Fixing DNSSEC mismatches after a nameserver move without taking the domain down again
• Adding DNS and mail-flow monitoring so the next zone edit doesn’t go unnoticed for a week

Send me the output of dig +trace MX yourdomain.com and a rejection message from a sender, and I can usually tell you where it’s breaking before we’ve agreed on anything.

The post Zoho Mail Not Receiving Email? A DNS Troubleshooting Guide appeared first on John Nessime.

By the time that ticket lands, the leak already happened. You can patch the query in an hour. What you cannot do is tell the affected garage how many other reports were wrong, how long it had been wrong, or whether anyone downloaded a CSV. There’s no log that answers those questions, because nothing ever denied anything. The database happily returned the rows. The API happily serialised them.

That’s the shape of the problem. Tenant isolation on AWS is not really about writing careful code. It’s about arranging things so that careless code fails loudly instead of quietly returning somebody else’s data.

This post walks through building a multi-tenant workshop management platform on AWS with isolation baked in from the first commit: where tenant context comes from, how the boundary changes shape per storage service, what the control plane has to own, and how you prove any of it works. The example is a shop management system, with job cards, vehicle histories, parts inventory and technician timesheets, but the patterns apply to any vertical SaaS product where one customer’s records must never touch another’s.

The failure mode that stays invisible

Most multi-tenant systems start with a tenant_id column and a convention: every query filters on it. That works right up until it doesn’t.

The convention breaks in ordinary ways. Someone adds a reporting endpoint and copies a query from a script that ran as an admin. A new join pulls in a table nobody remembered to filter. An ORM lazy-loads a relationship and the filter lives on the parent, not the child. A background job that recalculates parts margins runs without any tenant in scope at all, because it processes everything.

None of these throw. That’s the whole issue. A missing authorisation check produces a 403 you’ll notice in staging. A missing tenant filter produces a 200 with too many rows, and 200s don’t page anyone.

The fix is not more discipline. It’s moving the filter somewhere the application cannot forget it: into IAM, into the database engine, or both. AWS makes this point directly in its own SaaS guidance, and it’s the right one. If your only defence against cross-tenant reads is that developers remember, you don’t have a defence, you have a habit.

Pick the isolation model before you write code

There are three shapes, and they’re usually described as pool, silo and bridge.

• Pool. Every tenant shares the same tables, buckets and compute. Cheapest to run, cheapest to deploy, and the model where isolation has to be enforced explicitly because nothing physical separates anyone.
• Silo. Each tenant gets dedicated resources: its own database, its own bucket, sometimes its own account. Isolation is close to free, operations are not. Migrations, deploys and monitoring all multiply by tenant count.
• Bridge. Mixed. Shared compute, dedicated storage, or pooled for the standard tier and siloed for the enterprise tier that asked hard questions in procurement.

For a workshop platform, most independent garages will be small, and pooling is the only sane starting point. The trap is treating that as permanent. Sooner or later a dealer group with forty sites will ask for a dedicated database, and if you have not left room for a per-tenant routing decision, retrofitting it means rewriting your data access layer.

What I’d actually do: build pooled, but put the storage target behind a resolver from day one. A function that takes a tenant ID and returns a connection, a table name or a bucket prefix. When the first silo tenant arrives, you change the resolver, not four hundred call sites.

Where tenant context comes from

This is the part people get subtly wrong, and it undermines everything downstream. The tenant identifier must come from the authenticated identity, never from the request. Not a header, not a query parameter, not a field in the JSON body. If a client can influence the tenant ID, your isolation model is decoration.

In practice that means a custom claim in the token your identity provider issues. Amazon Cognito can carry a custom attribute for this, and so can any external IdP you federate with. The API layer reads the claim, and from that point the tenant is a fact about the caller rather than an input to the call.

Then you push that fact down into AWS itself using a session tag. When your service assumes a role, it attaches the tenant as a tag on the session. Every subsequent AWS API call made with those credentials carries the tag in the request context, where IAM policies can reference it as aws:PrincipalTag.

import boto3

def scoped_session(tenant_id, role_arn):
    sts = boto3.client("sts")
    resp = sts.assume_role(
        RoleArn=role_arn,
        RoleSessionName=f"workshop-{tenant_id}",
        Tags=[{"Key": "TenantID", "Value": tenant_id}],
    )
    c = resp["Credentials"]
    return boto3.Session(
        aws_access_key_id=c["AccessKeyId"],
        aws_secret_access_key=c["SecretAccessKey"],
        aws_session_token=c["SessionToken"],
    )

Two things make this work, and both are easy to miss. The role’s trust policy has to allow sts:TagSession alongside the assume-role action, or the call fails. And the calling principal needs permission to pass that tag. Get either wrong and you’ll spend an afternoon reading an error that sounds like it’s about the role rather than the tag.

One session tag, one role, one policy. That’s attribute-based access control, and it’s the reason ABAC scales where a role per tenant does not. Roles are a finite resource in an AWS account. Tags are not.

Tenant isolation on AWS changes shape per service

There is no single isolation control. Each service exposes a different lever, and you have to learn every one your architecture touches.

DynamoDB: the partition key is the boundary

If job cards live in DynamoDB with the tenant ID as the partition key, IAM can pin every read and write to that key. The condition key is dynamodb:LeadingKeys.

{
  "Effect": "Allow",
  "Action": [
    "dynamodb:GetItem",
    "dynamodb:PutItem",
    "dynamodb:UpdateItem",
    "dynamodb:Query"
  ],
  "Resource": "arn:aws:dynamodb:REGION:ACCOUNT:table/JobCards",
  "Condition": {
    "ForAllValues:StringEquals": {
      "dynamodb:LeadingKeys": ["${aws:PrincipalTag/TenantID}"]
    }
  }
}

Now a Query that omits the tenant partition key doesn’t return other tenants’ job cards. It gets denied. That’s the behaviour you want: loud, logged, and impossible to miss in CloudTrail.

Two caveats worth knowing before you commit. Scan operations don’t have a leading key to constrain, so granting dynamodb:Scan alongside this condition undermines the whole arrangement. And global secondary indexes have their own key structure, so an index whose partition key isn’t the tenant needs separate thought. Design the access patterns so that no query ever needs to look across tenants, and this stops being a problem.

S3: two surfaces, not one

Vehicle photos, inspection PDFs and signed job sheets go to S3 under a per-tenant prefix. The mistake is scoping only the object actions and leaving ListBucket open, which lets a tenant enumerate every other garage’s filenames even without reading them. Filenames leak plenty: customer names, registration plates, invoice numbers.

Object actions are scoped through the resource ARN. Listing is scoped through the s3:prefix request condition. You need both statements, because they protect different operations.

[
  {
    "Effect": "Allow",
    "Action": "s3:ListBucket",
    "Resource": "arn:aws:s3:::workshop-tenant-files",
    "Condition": {
      "StringLike": {
        "s3:prefix": ["${aws:PrincipalTag/TenantID}/*"]
      }
    }
  },
  {
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:PutObject"],
    "Resource": "arn:aws:s3:::workshop-tenant-files/${aws:PrincipalTag/TenantID}/*"
  }
]

The policy variable in the resource ARN is doing real work there. One policy, every tenant, no template rendering at request time.

Aurora PostgreSQL: row-level security, and the trap in it

Relational data is where most workshop platforms actually live, because job cards, parts lines and labour rates are relational. IAM cannot see inside a table, so the boundary moves into PostgreSQL itself via row-level security.

ALTER TABLE job_cards ENABLE ROW LEVEL SECURITY;
ALTER TABLE job_cards FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON job_cards
  USING (tenant_id = current_setting('app.tenant_id', true))
  WITH CHECK (tenant_id = current_setting('app.tenant_id', true));

USING controls which rows are visible to reads, updates and deletes. WITH CHECK controls what can be written. Without the second clause, a tenant can read only its own rows but insert a row stamped with someone else’s tenant ID. The first protects the read path, the second protects the write path, and you want both.

Now the trap, and it’s the one that gives teams false confidence. PostgreSQL superusers and roles carrying BYPASSRLS ignore row security entirely, and by default so does the table owner. If your application connects as the same role that ran the migrations, your policies are not in effect and your tests pass anyway. That’s why FORCE ROW LEVEL SECURITY is in the snippet above, and why the application should connect as a dedicated non-owner, non-superuser role.

The second trap is connection reuse. Set the tenant with SET LOCAL inside a transaction, or with set_config using the transaction-local flag. Plain SET persists for the life of the connection, and a pooled connection outlives the request. That’s how one garage’s context ends up serving the next garage’s query.

Test row-level security as the application role, against a pooled connection, with at least two tenants in the table. Testing as the owner tells you nothing.

Compute isolation is a different question

Data isolation stops one tenant reading another’s records. It does nothing about one tenant consuming everyone’s capacity.

Workshop platforms have a specific version of this. End of month, every garage runs its invoicing and MOT reminder batch at roughly the same time. A dealer group importing three years of service history will happily saturate a shared worker pool while forty independents wait for their job cards to load.

Levers worth knowing, roughly in order of how much they cost you:

• API Gateway usage plans, keyed per tenant, to cap request rates at the edge before anything expensive runs.
• Separate queues, or at minimum separate consumer concurrency, for bulk imports versus interactive requests. Bulk work should never share a lane with a screen someone is waiting on.
• Reserved or provisioned concurrency on the Lambda functions serving interactive paths, so a batch surge cannot starve them.
• Dedicated compute for premium tenants. This is silo by another name, and it’s the honest answer when a customer’s load profile genuinely doesn’t fit the pool.

Be honest with yourself about which problem you’re solving. Throttling is not isolation. It limits blast radius, it doesn’t create a boundary.

What the control plane has to own

Separate the control plane from the application plane early. The control plane manages tenants; the application plane serves them. Mixing the two is how a bug in the onboarding flow ends up with production credentials.

Onboarding a new workshop is a sequence, and it should be a single idempotent workflow rather than a checklist someone follows:

1. Generate a non-guessable tenant identifier. Lowercase alphanumeric, no customer name in it, because it will end up inside resource ARNs and key prefixes.
2. Write the tenant record: tier, isolation model, storage target, status.
3. Provision identity. User pool group or IdP mapping, with the tenant claim wired in.
4. Provision storage. For a pooled tenant that’s a prefix and a seeded row. For a siloed one it’s real infrastructure, which is why this step must be asynchronous.
5. Apply tags used for cost allocation and reporting.
6. Run a verification step that proves the new tenant can reach its own data and cannot reach a canary tenant’s data.

Step six is the one teams skip. It’s also the only step that tells you the previous five worked.

On per-tenant cost: activated cost allocation tags attribute anything that is a distinct tagged resource, which covers siloed tenants nicely. Pooled resources will not split by themselves, because a shared table doesn’t know which garage caused which read. If you need per-tenant margin, emit consumption as a metric dimension from the application: request counts, storage bytes, document pages processed. Tools like Vantage or CloudZero can allocate shared spend afterwards, but only from the signal you produce. Nothing recovers attribution you never recorded.

The AWS SaaS Builder Toolkit is worth a look here. It codifies control plane concepts as CDK constructs and will save you real time on onboarding plumbing. Read its own guidance first: the project describes itself as sample code, and expects you to review security fit before production. That’s a fair description rather than a warning label, but treat it as a starting point, not a finished platform.

Proving isolation actually holds

An isolation model you haven’t tried to break is a design document, not a control.

The tests that earn their keep are negative ones. Seed two tenants. Authenticate as the first. Then deliberately do the wrong thing: request the second tenant’s job card by ID, list the second tenant’s S3 prefix, run a query with the session variable set to the wrong value. Every one of those should fail, and the test should assert on the failure.

• Run the cross-tenant suite on every pull request, not nightly. It’s the regression that matters most and the one most likely to be introduced by an innocent refactor.
• Use the IAM policy simulator to check a policy change before it ships, particularly when someone widens a resource ARN.
• Turn on IAM Access Analyzer so external access grants surface without anyone having to notice them.
• Audit the database on a schedule: tables with RLS enabled but no policy attached, tables missing FORCE, application roles that have quietly acquired ownership or BYPASSRLS during an incident.
• Alert on AccessDenied volume per tenant. A spike is either a bug you introduced or someone probing, and both are worth knowing about.

Keep a permanent canary tenant in every environment, including production, holding nothing but synthetic data. Every negative test targets it. It costs almost nothing and it means your isolation tests never need real customer records.

Troubleshooting the failures you’ll actually hit

AccessDenied on a policy that looks correct

Almost always the session tag isn’t present. If the tag is missing from the request context, the condition can’t match and the statement doesn’t apply. Call sts:GetCallerIdentity with the scoped credentials and confirm you’re on the assumed role you think you are, then check CloudTrail for the AssumeRole event and look at whether the tag was actually passed. Tag keys are case sensitive, and TenantId is not TenantID.

Queries return zero rows instead of the right rows

Classic RLS symptom. The session variable is unset, so the policy predicate compares against null and nothing matches. Check current_setting inside the same transaction as the query, not in a separate connection from your SQL client. Empty results are the safe failure here, which is exactly why they’re easy to misread as a data problem.

One tenant intermittently sees another’s data

Intermittent means state reuse. Look at connection pooling first, then at any per-request context stored in thread-local or async-local storage that isn’t reset when the request finishes. If you’re using a proxy in front of the database, understand how it handles session state, because some proxies pin a connection to a client once session-level settings are detected, which changes the behaviour you tested against.

Isolation works in the API but not in background jobs

Because the job has no request, so it has no token, so it has no tenant. Whatever runs asynchronously needs the tenant carried on the message and a session assumed per tenant when the work is processed. A worker that loops over all tenants with admin credentials is the single most common place isolation quietly stops applying.

Session tags don’t survive a second AssumeRole

Session tags are not automatically carried forward when you chain roles unless they were marked transitive. If your architecture hops through more than one role, this is where the tenant context evaporates. Flatten the chain if you can; if you can’t, mark the tag transitive deliberately and document why.

Common mistakes

• Taking the tenant ID from a request header or body instead of the authenticated token.
• Using the customer’s name or a sequential integer as the tenant identifier, then putting it in bucket prefixes where it becomes both guessable and enumerable.
• Scoping S3 object actions but leaving bucket listing wide open.
• Running the application as the PostgreSQL table owner, so RLS is enabled and silently inert.
• Writing a USING clause with no WITH CHECK, leaving the write path open.
• Creating one IAM role per tenant and discovering the account ceiling somewhere around the point the business gets interesting.
• Assuming cost allocation tags will attribute pooled spend. They won’t, and by the time you need the numbers the history is gone.
• Testing isolation only through the UI, where the frontend is already sending the right tenant every time.

Best practices worth the effort

• Derive tenant context from the token, propagate it as a session tag, and never let application code choose it.
• Enforce the boundary at the layer below your code: IAM conditions for AWS resources, RLS for relational rows.
• Put storage targets behind a resolver so moving a tenant from pool to silo is a configuration change.
• Make onboarding one idempotent workflow that ends in a verification step.
• Emit tenant as a dimension on logs and metrics from the start, so cost and performance questions stay answerable.
• Keep a canary tenant and run cross-tenant negative tests in CI on every change.
• Put a WAF or edge layer such as Cloudflare in front of tenant subdomains, and keep tenant routing decisions out of the origin application where you can.

Frequently asked questions

Is a shared database with a tenant_id column ever acceptable?

Yes, provided the column is enforced by the engine rather than by convention. A shared table with row-level security, forced, queried by a non-owner role, is a real boundary. A shared table where every query is expected to include the filter is not. The schema is the same; the guarantee is completely different.

ABAC or dynamically generated IAM policies?

ABAC for most cases. One role, one policy, tenant supplied per session, and nothing grows as you add customers. Dynamic policy generation, sometimes called a token vending machine, earns its place when a single policy genuinely cannot express the rule, such as when per-tenant resource names have to be injected rather than a key prefix. The cost is that you now own the correctness of a policy generator, plus per-request latency, and session policies have a size ceiling you can hit.

Should each tenant get its own AWS account?

It’s the strongest boundary available and the most expensive to operate. For a workshop platform serving independent garages it’s overkill. It becomes reasonable when a customer’s contract, regulator or data residency requirement makes shared infrastructure a non-starter, and at that point you’re pricing it as a premium tier rather than absorbing it.

How do I handle a user who works at two workshops?

Model it as one identity with multiple tenant memberships and an explicit active tenant per session, rather than a token carrying a list. The active tenant becomes the session tag. Switching workshops means a new session, which is exactly the behaviour you want because it makes the switch visible in your audit trail.

Does row-level security hurt query performance?

It adds a predicate the planner has to satisfy, so the answer depends on your indexes. Index the tenant column, and index it as the leading column of composite indexes that support your common filters. Compare plans as the application role before and after enabling policies, because a plan captured as the owner may not reflect what the application actually runs.

Where does application-level authorisation fit?

Alongside, not instead. Tenant isolation answers “which organisation’s data is this”. Authorisation answers “may this technician void an invoice”. Different questions, different layers. A policy engine such as Amazon Verified Permissions handles the second cleanly, and keeping them separate stops role logic creeping into your isolation boundary.

Can I retrofit isolation onto a platform that already has tenants?

You can, and it’s tedious rather than impossible. Enforce at the database first, because that’s where the leak actually happens: table by table, FORCE enabled, application moved to a non-owner role. Then move the tenant ID out of request payloads and into the token. Add IAM conditions last, since they’re the least likely source of a live leak. Expect to find at least one background job with no tenant scope at all.

The one thing to remember

Tenant isolation on AWS works when the boundary sits below your application code, in a layer that denies rather than trusts. IAM conditions on session tags for AWS resources, forced row-level security for relational data, and a control plane that verifies a new tenant is properly fenced before anyone logs in.

Retrofitting that onto a running multi-tenant platform is possible but grim, because you’re doing it while real workshops have real data in the system. Doing it on day one costs a week. That’s the entire trade, and it’s not a close call.

Need a second pair of eyes on your multi-tenant architecture?

Most of the isolation problems I see are not exotic. They’re a missing WITH CHECK clause, an application connecting as the table owner, or a background job nobody scoped. Things I can help with:

• Reviewing an existing multi-tenant design and finding where the boundary is enforced by convention rather than by the platform.
• Implementing ABAC with STS session tags across DynamoDB, S3 and Aurora, including the trust policy wiring that trips people up.
• Setting up PostgreSQL row-level security correctly, with forced policies, a dedicated application role and pooling that doesn’t leak session state.
• Building a tenant onboarding workflow that provisions identity, storage and tagging idempotently and verifies itself.
• Writing the cross-tenant negative test suite and wiring it into CI so isolation regressions fail the build.
• Adding per-tenant usage metering so cost, performance and tier decisions rest on data instead of guesses.

Send me a policy document, an RLS definition or a CloudTrail AccessDenied event and I’ll tell you what it’s actually enforcing.

The post Tenant Isolation on AWS: Building a Multi-Tenant Workshop Platform That Doesn’t Leak appeared first on John Nessime.