Moving the files is the solved part. Which bucket they land in, and what sits in front of that bucket, decides whether you spend almost nothing or spend more than the server you were trying to shrink.

Cloudflare R2 vs Amazon S3 usually gets reduced to one sentence: R2 does not charge for data transfer out, S3 does, therefore R2. That is wrong in both directions, and understanding why is most of the value here. This post covers the failure mode that shows up after the migration rather than during it, an honest profile of each service, the levers that actually move the number, and a procedure you can run against your own traffic in an afternoon.

Free egress is not free serving

R2 charges nothing to move bytes out, on any storage class. It still charges for the read. Every GetObject and every HeadObject is a Class B operation, and a media library made of thousands of small files racks those up fast. Cloudflare’s own asset hosting example in the R2 pricing docs makes the point: storage falls inside the free tier, writes fall inside the free tier, and the whole bill is read operations.

So the meter is not gigabytes. The meter is requests that reach the bucket, which means the variable deciding your bill on either platform is cache hit ratio.

This is where offloading setups go wrong without anyone noticing. The plugin rewrites image URLs to point at the bucket, images render, pages look fine, and every request for every thumbnail travels all the way to the origin and gets billed. Nothing breaks. Nothing alerts. The number just grows with traffic.

There is a sharper version on the R2 side. A new bucket gives you a public URL on an r2.dev subdomain, which is convenient and tempting. Cloudflare documents it as a testing endpoint with a variable rate limit. Push production traffic through it and requests get throttled with 429 Too Many Requests, throughput can be throttled too, and you get no cache, no WAF, no bot management. Those only exist once the bucket sits behind a custom domain you control. That is the invisible failure: not a broken image, a working setup billed on every request and rate limited under load.

Cloudflare R2: where it wins and where it hurts

Where R2 wins

• Egress really is zero. Not zero within a partner network, not zero up to a ratio of stored data. Zero, on both storage classes, through the S3 API and through Workers.
• A custom domain puts it behind the CDN. One managed CNAME and reads start being absorbed at the edge instead of hitting the bucket. No second product to buy and wire up.
• Deletes are free operations, along with aborting a multipart upload. Cleaning up a messy library costs nothing.
• Unauthorized requests are not billed. A caller without permission gets a 401 and you are not charged, which matters once someone starts hammering your bucket path.
• Migration tooling is free to use. Super Slurper copies a bucket across in bulk, Sippy migrates lazily on first miss. You pay only for the operations they perform against R2.

Where R2 hurts

• S3 compatible is not S3 identical. Cloudflare publishes a table of which operations are implemented and which are not. Real tooling has broken on the gaps: when AWS shipped SDKs that enabled CRC32 checksums by default, R2 rejected the header until the mismatch was resolved.
• There is no cold archive tier. Infrequent Access is as cold as it gets, and it carries a minimum storage duration, a retrieval fee, and doubled operation rates. For an active library that is the wrong tier, not a saving.
• Data lives in one primary location. Location hints are best effort rather than a region guarantee, and are honored only the first time a bucket with that name is created. Read performance away from that location comes from the edge cache, not from replicas.
• Plugin support is thinner. WP Offload Media, the long-standing WordPress option, officially lists Amazon S3, DigitalOcean Spaces, and Google Cloud Storage. R2 users generally land on Media Cloud, Advanced Media Offloader, Next3 Offload, or the S3-Uploads route driven from WP-CLI.

Amazon S3: where it wins and where it hurts

Where S3 wins

• Origin transfer to CloudFront is not billed. This quietly demolishes the usual comparison. AWS waives data transfer from an AWS origin to CloudFront, so a properly built S3 media stack never pays the direct-to-internet rate on cached traffic.
• The CloudFront free allowance is permanent, covering a monthly volume of data transfer out and requests rather than expiring after twelve months. Small sites can sit inside it indefinitely.
• Event driven derivatives are mature. Upload fires a Lambda that generates thumbnails, strips metadata, converts formats, writes the result back. Hard to beat if your media pipeline does real work.
• A full storage ladder with real lifecycle transitions, down to deep archive. If a large chunk of your library is genuinely cold, S3 has somewhere cheap to park it.
• Governance depth. Object Lock, versioning, replication, IAM condition keys, CloudTrail data events. When an auditor asks for a control, S3 usually has a named feature for it.

Where S3 hurts

• The default configuration is the expensive one. Point image URLs at the bucket endpoint and every byte leaves at the internet rate with nothing cached in front of it. The cheap path has to be built deliberately.
• Four meters instead of two: storage, bucket requests, CDN requests, CDN data transfer, plus per-feature charges. Forecasting takes real effort.
• Geography changes the rate. Delivery costs more per gigabyte to some regions than others, so an audience shift moves your bill without you changing anything.
• Leaving costs money. Copying a large library out is billed transfer, once, in a lump. Usually small against ongoing savings, but it is a real number somebody has to approve.

How to decide between Cloudflare R2 vs Amazon S3

Four levers move the outcome. Everything else is noise.

Cache hit ratio

The dominant term on both platforms, and the one people skip. At a high hit ratio the origin barely gets touched and both bills collapse toward the storage line. At a low hit ratio, R2 bills read operations and S3 bills origin GETs, and both climb with traffic. Media should be trivially cacheable because the files never change. If yours is not caching, that is a configuration problem worth fixing before you pick a vendor, because fixing it changes the answer.

Object count, not library size

WordPress does not store one file per upload. It stores the original plus every registered image size, and themes and page builders happily register more. Ten thousand uploads can easily be sixty thousand objects, each one a write on migration and a read on a cache miss. Count them before you model anything:

cd /path/to/wordpress

# Objects you are about to create in the bucket
find wp-content/uploads -type f | wc -l

# Bytes
du -sh wp-content/uploads

# How much of that count is derivative sizes rather than originals
find wp-content/uploads -type f -regextype posix-extended 
  -regex '.*-[0-9]+x[0-9]+.(jpe?g|png|webp|avif)$' | wc -l

That last number is usually the surprise. If most of your object count is thumbnails, trimming unused registered sizes before migrating is the cheapest optimization available, and it shrinks writes, reads, and storage at once.

How cold the library actually is

Most libraries are a long tail. A small set of recent files takes nearly all the traffic while the rest sits untouched for years. If that tail is large, S3 has somewhere genuinely cheap to put it and R2 does not. Be careful with R2’s Infrequent Access class here: it lowers the storage rate but raises both operation rates, adds a retrieval fee, and enforces a minimum duration whether you keep the object or not. It fits write-once, read-almost-never data, not a library anything still links to.

What the rest of your stack already is

If your application already runs in AWS and generates derivatives with Lambda on upload, moving to R2 means rebuilding that pipeline against Workers or R2 event notifications. That is real work you do not get paid for. If your site is a VPS from somewhere like InterServer or Contabo with Cloudflare already in front of it, R2 is nearly free work: the DNS is there, the cache is there, and the bucket slots in behind a subdomain you already control.

Setup details that change the answer

A few configuration choices matter more than the vendor choice. Get these wrong and the cheaper platform produces the bigger bill.

Serve from a custom domain, always

On R2 this is the difference between a cached, protected asset host and a throttled test endpoint. Attach one from the dashboard or from Wrangler:

npx wrangler r2 bucket domain add my-media-bucket 
  --domain=cdn.example.com 
  --zone-id=<YOUR_ZONE_ID>

npx wrangler r2 bucket domain list my-media-bucket

Then disable public access on the r2.dev subdomain, or you have left a second uncached, unprotected door into the same objects.

Verify the cache instead of assuming it

Assumptions about caching are where the money leaks. Ask the edge directly: request the same object twice and read the headers.

# Cloudflare in front of R2
curl -sI https://cdn.example.com/2024/07/photo-1024x768.jpg 
  | grep -iE 'cf-cache-status|cache-control|age'

# CloudFront in front of S3
curl -sI https://cdn.example.com/2024/07/photo-1024x768.jpg 
  | grep -iE 'x-cache|cache-control|age'

First request, expect a miss. Second, expect a hit. If the second still reports a miss, or reports a status meaning the response was never eligible for caching, every image view is costing you an origin read. Fix that before you compare anything.

Long cache lifetimes and versioned filenames

Uploaded media is immutable in practice. Nobody edits the bytes of photo-1024x768.jpg, they upload a new file. So cache lifetimes should be long and the object name should change when the content does. If you find yourself purging image caches regularly, the real problem is that your filenames are not versioned.

On the Cloudflare side, Smart Tiered Cache is worth enabling for R2 origins. It routes edge misses through an upper tier data center close to your bucket instead of letting every edge location fetch independently, which cuts the number of requests reaching R2 at all.

Clean up failed uploads

Large uploads use multipart. Interrupted ones leave orphaned parts that occupy billed storage and do not show up in a normal object listing. Both platforms support lifecycle rules for this, and on R2 you can manage them from Wrangler:

npx wrangler r2 bucket lifecycle list my-media-bucket

npx wrangler r2 bucket lifecycle add my-media-bucket 
  --name=expire-temp 
  --prefix=tmp/ 
  --expire-days=30

Configure an abort rule for incomplete multipart uploads on day one, whichever platform you land on. It is the most common source of storage you pay for and cannot see.

A decision procedure you can run this afternoon

1. Measure your current cache hit ratio for image paths, from analytics or a sampled read of access logs. Everything downstream depends on this number.
2. Count objects, not gigabytes. Run the find commands above and separate originals from derivative sizes.
3. Estimate monthly origin reads as total image requests multiplied by the miss rate. That, not your bandwidth, is what you feed into either pricing calculator.
4. Treat writes as a one-off migration spike plus a modest steady rate. Migration is usually the largest write event the bucket ever sees.
5. Price both against current published rates on the same day, using Cloudflare’s R2 calculator and the AWS pricing calculator. Rates move. Do not trust a number copied from a blog post, including this one.
6. Test with a prefix before committing. Offload one year of uploads, point the site at it, watch cache status headers and the operations dashboard for a week. A week of real traffic beats any spreadsheet.

If it comes out close, take R2. Not because it is cheaper in that scenario, but because it has fewer meters to reason about, and a simpler operational model is worth something on the day something breaks.

Arguments that don’t survive contact

“S3 charges egress, R2 doesn’t, so R2 always wins”

Only true against the worst possible S3 setup. Transfer from an S3 bucket to CloudFront is not billed, so a properly built S3 media stack pays CDN delivery rates on cache misses, not the direct-from-bucket rate. Add CloudFront’s permanent free allowance and a small site pays nothing on either platform. The gap opens at scale and on origin-heavy traffic, not on the first terabyte.

“R2 is a CDN”

R2 is object storage that can sit behind a CDN you already have. The caching, the WAF, the bot rules all come from the Cloudflare zone, and none of it applies to the development URL. If the bucket is not behind a custom domain on a zone you control, you have storage without delivery.

“S3 compatible means drop-in”

It means most tools work with a changed endpoint and a region value of auto. It does not mean every operation, header, and checksum behaves identically. Budget an afternoon for fighting an SDK default, and test uploads, multipart uploads, deletes, and signed URLs against the real bucket before cutting over.

“We’ll save money moving cold files to Infrequent Access”

Sometimes. The lower storage rate arrives with higher operation rates, a retrieval fee, and a minimum duration. If files are cold enough to justify that, they are probably cold enough to belong in a real archive tier, which R2 does not have. Model it against your read pattern rather than assuming a cheaper per-gigabyte number is a cheaper bill.

Frequently asked questions

Is Cloudflare R2 cheaper than S3 for a WordPress media library?

Usually, once traffic is meaningful and the library is served publicly. For a small site behind a well configured CDN, both can land near zero. R2’s advantage grows with egress volume and with the number of requests that miss cache, which is why measuring hit ratio comes first.

Can I use the same WordPress plugin for both?

Several plugins speak the S3 API and support both, including Media Cloud, Advanced Media Offloader, and Next3 Offload. WP Offload Media officially lists Amazon S3, DigitalOcean Spaces, and Google Cloud Storage, so check current provider support before assuming a swap is free. Whichever you pick, confirm it rewrites existing URLs and not only new uploads.

How do I migrate an existing S3 bucket to R2 without downtime?

Cloudflare offers two paths. Super Slurper copies everything in bulk. Sippy migrates incrementally, pulling an object from the source the first time it is requested and serving from R2 afterward. Both are free to use, you pay for the operations they perform against R2, and your source bucket may charge you for the reads.

Do I still need a CDN in front of R2?

You need a custom domain on a Cloudflare zone, which is what puts the CDN in front of it. That is not optional for production. The development URL is rate limited by design and gets no cache, WAF, or bot management.

Can I keep S3 and just put Cloudflare in front of it?

You can, and it does cut origin reads. Be aware that cache misses then pull from S3 across the public internet, billed as ordinary S3 egress, unlike an S3 origin sitting behind CloudFront. Reasonable as an interim step, rarely the cheapest end state.

What about Backblaze B2, Wasabi, or DigitalOcean Spaces?

All viable and worth pricing if you are already modelling. B2 competes on raw storage, Wasabi sells predictability, Spaces is convenient when your droplets are already there. The analysis transfers directly: find the meters, find your cache hit ratio, multiply.

How do I monitor the bill after migrating?

Watch operations and cache hit ratio, not storage. Storage grows slowly and predictably. Operations track traffic and configuration mistakes, which is where the unpleasant surprises live. A Grafana Cloud dashboard or a scheduled pull from the provider’s usage API is enough; the point is that somebody looks weekly rather than at invoice time.

Conclusion

If you take one thing from this Cloudflare R2 vs Amazon S3 comparison, take this: the vendor choice is second order. What decides your media offloading bill is how many requests reach the bucket, and that is a function of your cache configuration, your object count, and whether you put a real domain in front of the storage.

Fix the cache first. Count the objects. Then price both, on the same day, against your own numbers. Most of the time R2 comes out ahead for public media, S3 comes out ahead when the library is entangled with AWS services or needs a genuine archive tier, and the difference is smaller than the internet suggests.

Need help moving a media library without breaking it?

Media offloading looks like a plugin install and turns into a URL rewriting, caching, and permissions problem. I work with teams on the parts that are easy to get subtly wrong:

• Modelling R2 against S3 using your real object counts, request volumes, and cache hit ratio instead of a generic calculator
• Planning and running the migration: bulk or incremental copy, URL rewriting, and a rollback path if the cutover misbehaves
• Custom domain, cache rule, and header configuration so origin reads collapse instead of tracking your traffic
• Bucket permissions, CORS, signed URL flows for private downloads, and lifecycle rules for orphaned multipart uploads
• Trimming registered image sizes and derivative sprawl before migration so you stop paying to store thumbnails nothing links to
• Dashboards for operations, cache hit ratio, and storage growth so the bill stops being a monthly surprise

Send me a curl -I of one of your image URLs and a rough object count, and I can usually tell you quickly whether you have a pricing problem or a caching problem.

The post Cloudflare R2 vs Amazon S3 for Media Offloading: Where the Bill Actually Comes From appeared first on John Nessime.

That is the failure mode worth internalizing before anything else: Amazon Macie PII detection reports on what it managed to read, not on what is there. Objects it never opened produce no findings, and no findings looks identical to a clean result. The gap between “we scanned this” and “we found nothing” is where most Macie deployments quietly fail their first audit.

This post covers where that gap comes from and how to close it: the eligibility chain every object goes through, the four distinct reasons an object gets skipped, how to tune identifiers so the findings are worth reading, what actually drives the bill, and how to route results somewhere a human will act on them.

The eligibility chain behind Amazon Macie PII detection

Every object passes the same gates before a byte gets inspected. The order matters, because each gate fails differently and needs a different fix.

1. Is it in a general purpose S3 bucket? Directory buckets are out of scope entirely.
2. Is the storage class supported? Standard, Standard-IA, One Zone-IA, Intelligent-Tiering, Glacier Instant Retrieval and Reduced Redundancy are in. Glacier Deep Archive and S3 Express One Zone are not.
3. Does the object key carry a recognized file extension? This is a string check on the name, not a content sniff.
4. Can Macie retrieve and decrypt it, given the bucket policy, object ACL and encryption key?
5. Does the content parse, and stay inside the per-file quotas?
6. Do the active data identifiers match anything in it?

Only the last gate produces a finding. Everything before it produces silence. Macie does record why each object was skipped, but that lives in coverage data and object samples, not on the findings page most people look at.

One detail explains a lot of confusion: when you first enable automated sensitive data discovery, every bucket gets a sensitivity score of 50 and the label Not yet analyzed. A bucket whose permissions block Macie stays there permanently. It never turns red. It sits mid-list, looking unremarkable next to buckets that genuinely were analyzed.

Gap one: objects Macie never opened

Macie calls these unclassifiable. They fail the storage class or extension check, so no retrieval is attempted. This is the largest source of false confidence I’ve seen, and the cheapest to fix.

Missing file extensions

Classifiability comes from the file name extension. A file full of customer records named part-00003-a4f9 is invisible. The same bytes named part-00003-a4f9.json get inspected and produce findings.

This bites hardest on streaming ingestion. Amazon Data Firehose writes to S3 without appending an extension unless compression, format conversion, or the explicit file extension setting adds one. An uncompressed JSON stream lands as a tree of extensionless objects. Athena and Glue read them fine, because they infer format from the table definition. Macie skips every one.

Fix it upstream: set the S3 file extension on the delivery configuration, or enable GZIP compression or Parquet conversion, both of which append a recognized extension. If you can’t change the producer, the fallback is a copy-and-rename step into a scan prefix, which costs you storage plus a pipeline to maintain. Fixing the producer is almost always cheaper.

Unsupported storage classes

A lifecycle rule transitioning old exports to Glacier Deep Archive quietly removes them from scope. That may be the right storage decision, but it needs to be a conscious one, because archived customer data is still customer data when a regulator asks.

Macie exposes these counts per bucket, split by cause. This is the first query I run against a new account:

# Buckets where Macie can't classify objects, broken down by cause
aws macie2 describe-buckets 
  --query 'buckets[?unclassifiableObjectCount.total > `0`].{
      bucket: bucketName,
      classifiable: classifiableObjectCount,
      skipped_total: unclassifiableObjectCount.total,
      skipped_extension: unclassifiableObjectCount.fileType,
      skipped_storage_class: unclassifiableObjectCount.storageClass
    }' 
  --output table

Read the ratio, not the raw number. Forty thousand classifiable objects and twelve skipped is fine. Three classifiable and four hundred thousand skipped is a reporting artifact pretending to be a scan result, and that bucket’s sensitivity score means nothing.

Gap two: objects it opened and couldn’t read

These are classification errors rather than eligibility failures. Macie selected the object, tried to fetch it, and was refused. Three causes, and they need three different people to fix them.

• Customer-provided keys (SSE-C). Macie cannot supply the key material, so it cannot retrieve the object. No permissions fix exists. Re-encrypt with S3 managed or KMS keys if you want coverage.
• KMS key policy. For customer managed keys, the key policy must allow the Macie service-linked role, AWSServiceRoleForAmazonMacie, to decrypt. Cross-account buckets need the key owner to grant it, not the bucket owner. AWS publishes a permission analyzer script in the aws-samples/amazon-macie-scripts repository that enumerates every key Macie needs and generates the CLI commands to fix them.
• Restrictive bucket policies. An explicit Deny conditioned on source IP or VPC endpoint blocks Macie along with everything else. The working pattern excludes the service-linked role ARN from the deny using the aws:PrincipalArn condition key.

To see which objects Macie actually touched in a bucket, pull the object samples. It’s the closest thing to a scan log you get:

# Objects automated discovery selected, with per-object status
aws macie2 list-resource-profile-artifacts 
  --resource-arn arn:aws:s3:::your-bucket-name

Anything with a status of SKIPPED is a lead. Take the key, check its encryption settings in S3, and you usually have your answer inside a minute.

Gap three: it read the file and nothing matched

Macie ships managed data identifiers covering common PII, financial data and credentials across many countries. Automated discovery uses a recommended subset by default rather than all of them. Sensible, but it is a default, and it will miss things that matter to you.

Add the managed identifiers for your actual jurisdictions. If you hold records for customers in a country whose national ID identifier isn’t in the recommended set, you’re scanning for the wrong things with perfect efficiency. Pull the current list with aws macie2 list-managed-data-identifiers and compare it against where your customers live.

Write custom identifiers for what only you know is sensitive. Internal account numbers, case references, employee IDs. A custom identifier is a regex plus optional keywords and a proximity rule, and the keywords are what save you. A bare [0-9]{8} matches timestamps, order totals and row counts, and you drown. The same regex with keywords and a match distance only fires when the number sits near a word that gives it meaning.

Test the pattern before it goes near a job. This runs the criteria against sample text and returns match counts without creating anything:

# Dry-run a custom identifier before creating it
aws macie2 test-custom-data-identifier 
  --regex 'ACC-[0-9]{4}-[0-9]{4}' 
  --keywords account customer acct 
  --maximum-match-distance 30 
  --sample-text 'customer account ACC-4821-9930 updated; total 1234-5678'

Run it against real samples, including the awkward ones, then against a file you know is clean and confirm zero. A pattern you only tested on positives is a pattern you haven’t tested.

The mirror image is allow lists, which tell Macie to ignore specific text or patterns: your support inbox address, your published press contact, the seeded test records in everyone’s staging fixtures. Without them the same non-issues appear in every scan until the team stops reading findings. Allow lists accept predefined text or regex, and the per-account and per-job quotas are tight enough that you should curate rather than dump.

Gap four: quota truncation inside large files

Subtle, because the object is analyzed. Just not completely, and a partial result reports like a full one.

• Per-file size quotas differ by format. Exceed the quota for a type and Macie analyzes none of that file, not part of it.
• Archives have separate ceilings for nesting depth, extracted bytes and file count. If the metadata shows a breach up front, nothing is extracted. Cross a limit partway through and Macie stops, reporting only what it processed.
• JSON and JSON Lines have a nested depth limit. Past it, the file is not analyzed at all.
• Full names and mailing addresses cap out per file. After the cap Macie stops counting and stops reporting locations for that type, so the occurrence count is a floor, never a total.

Findings carry a status reason naming the limit that was hit, with values like FILE_EXCEEDS_SIZE_LIMIT, ARCHIVE_NESTING_LEVEL_OVER_LIMIT and JSON_NESTING_LEVEL_OVER_LIMIT. Filter on those before declaring a data lake of large compressed exports clean. Check current quota values in the Macie documentation rather than memorizing them, since they differ per format.

Automated discovery or a discovery job?

Automated sensitive data discovery evaluates your inventory daily and samples representative objects across as many buckets as it can, grouping by metadata like prefix, extension and last-modified date. It is breadth-first and it does not re-analyze unchanged objects. Its job is to tell you which buckets deserve attention. It is not designed to prove a bucket is clean, and reading it that way is the root of most Macie misunderstandings.

Sensitive data discovery jobs are the depth tool. You choose the buckets, scope by prefix, extension, size or object tag, choose the identifiers, and run once or on a schedule. This is what you point at the bucket automated discovery just flagged.

# One-time job over a single prefix, recommended identifiers, 30% sampling
aws macie2 create-classification-job 
  --job-type ONE_TIME 
  --name "exports-prefix-review" 
  --managed-data-identifier-selector RECOMMENDED 
  --sampling-percentage 30 
  --s3-job-definition '{
    "bucketDefinitions": [
      {"accountId": "111122223333", "buckets": ["your-bucket-name"]}
    ],
    "scoping": {
      "includes": {
        "and": [
          {"simpleScopeTerm": {
             "comparator": "STARTS_WITH",
             "key": "OBJECT_KEY",
             "values": ["exports/"]
          }}
        ]
      }
    }
  }'

One detail about --sampling-percentage that trips people up: it selects a random percentage of eligible objects and then analyzes each selected object completely. It does not read a fraction of each file. For a bucket where every object comes from one pipeline, 20 or 30 percent tells you what you need. For heterogeneous uploads, sampling is a coin flip and you want the full pass.

What actually drives the bill

Macie bills on three dimensions, and the one that surprises people is not the one they budget for.

• Bucket evaluation. Per general purpose bucket monitored, prorated daily, up to the account ceiling. Predictable, driven by bucket count rather than data volume.
• Object monitoring. Per object tracked while automated discovery is enabled. This is inventory bookkeeping, not content reading, and it runs whether or not anything gets scanned that day. On an account with a log archive holding hundreds of millions of tiny objects, it can dwarf the other two.
• Data inspected. Per GB actually analyzed, covering both automated discovery and jobs.

So excluding buckets from automated discovery is a genuine cost lever, not just noise reduction. Log archives, CloudTrail destinations, build artifact stores and backup targets are usually safe exclusions. Exclude for the right reason though: a backup bucket holding database dumps of your customer table is exactly what you want scanned, however boring its name sounds.

The three dimensions appear as distinct usage types in the Cost and Usage Report, so they split cleanly. A FinOps platform like Vantage or CloudZero, or just a Cost Explorer view grouped by usage type, tells you within a day whether object monitoring or inspection is driving the number. That changes the fix entirely: object monitoring is solved by scoping, inspection by sampling and scheduling.

Getting findings somewhere a human sees them

Findings that only exist in the Macie console get read during onboarding week and never again. Two things are worth wiring up on day one.

Publish findings outward. Macie emits findings to EventBridge and integrates with Security Hub. EventBridge earns its keep because you can filter on severity and finding type and route only what matters, whether that’s a Slack channel, a ticket, or a Lambda that tags the bucket. If you already centralize alerts in Grafana Cloud or similar, that path keeps sensitive data findings next to everything else on call sees.

Configure the discovery results repository. Findings tell you what was found and where. The detailed discovery results, including analysis logs for objects where nothing was found, need an S3 bucket and a KMS key configured to persist. Nothing is retained long term until you set this up, and “we scanned it and found nothing” is exactly the record an auditor asks for.

One operational note on the reveal feature, which retrieves sample occurrences so a human can confirm a finding is real. It needs a customer managed KMS key and it’s genuinely useful for triage. It also means someone is now looking at live customer PII, which belongs in your access model and audit trail. Same goes if an engineer downloads an object to investigate locally: that copy is unmanaged PII on a laptop and needs a secure deletion step, whether that’s a wipe tool from a vendor like O&O Software or a documented and enforced process. Findings triage has a habit of creating the exposure it was meant to prevent.

Troubleshooting a bucket that reports nothing

When Amazon Macie PII detection reports nothing for a bucket you have doubts about, work down this in order. Each step rules out one gate, and stopping at the first hit saves you the rest.

1. Check the sensitivity label. Not yet analyzed at score 50 means nothing was analyzed. Do not read that as clean.
2. Check the coverage page. Macie names the issue directly: access denied, invalid encryption, invalid KMS key, permission denied, or unclassifiable.
3. Compare classifiable and unclassifiable counts with the describe-buckets query above.
4. List a few object keys. Run aws s3 ls against a prefix and look for extensions. Extensionless keys explain the whole thing.
5. Check the storage class. A lifecycle rule may have moved everything somewhere Macie doesn’t read.
6. Check encryption. SSE-C is unfixable without re-encryption. A customer managed KMS key is fixable with a key policy change.
7. Check object samples with list-resource-profile-artifacts to see exactly what was selected and skipped.
8. Only now question the identifiers. Run a targeted job with a custom identifier for the data you expect and see whether it fires.

Common mistakes

• Treating a low sensitivity score as evidence of no PII. It reflects what was found and how much was analyzed, so low coverage produces a low score.
• Enabling Macie in one region and assuming account-wide coverage. Settings and results are per region, every time.
• Running default identifiers in a business operating outside the jurisdictions the recommended set covers.
• Building custom identifiers from regex alone, then abandoning Macie over the false positive volume.
• Skipping the discovery results repository, then having no evidence trail when someone asks what was scanned and when.
• Excluding buckets by name pattern without checking contents. Backup and export buckets are frequently the highest-risk ones you own.

Best practices

• Measure coverage before findings. A coverage report is the first deliverable of a Macie rollout, not an afterthought.
• Fix extensions at the producer rather than building a rename pipeline downstream.
• Run the KMS permission analyzer script once per account, then again whenever a new customer managed key appears.
• Use automated discovery for breadth and targeted jobs for depth, letting the first choose targets for the second.
• Test every custom identifier against both a positive and a negative sample before it reaches a job.
• Curate allow lists early. The cost of a noisy first month is a team that stops reading findings.
• Re-check coverage after any change to bucket policies, KMS keys, lifecycle rules or ingestion pipelines. All four silently remove data from scope.

Frequently asked questions

Can Amazon Macie scan anything other than S3?

No. Macie analyzes objects in S3 general purpose buckets only. The documented workaround is bringing data to it: export RDS or Aurora snapshots to S3 in Parquet, or export a DynamoDB table to S3, then run a discovery job against the export. That works, but the export is now a second copy of your sensitive data with its own encryption and deletion requirements.

Does Macie scan new objects as they land?

Not per object. Automated discovery evaluates inventory on a daily cycle and prioritizes objects that are new or recently changed, so fresh data moves up the queue rather than triggering an immediate scan. If you need scanning tied to arrival, drive a discovery job from S3 event notifications through EventBridge and accept that you own that orchestration.

Why does a bucket I know has PII show a low sensitivity score?

Almost always coverage rather than detection. Check whether the objects are classifiable, whether Macie can decrypt them, and whether the label reads Not yet analyzed. If Macie genuinely read them and found nothing, the next suspect is the identifier set, and a targeted job with a custom identifier will tell you in one run.

Do I need both automated discovery and discovery jobs?

For most teams, yes. Automated discovery gives you the map at predictable cost. Jobs give you proof for a specific bucket at a specific time, which is what compliance evidence actually requires. Running only jobs means you never discover the bucket nobody told you about.

How do I cut false positives without missing real PII?

In order of preference: add keywords and a proximity rule so matches need context; use allow lists for specific known-benign values like your published contact details and test fixtures; and only then remove managed identifiers that are consistently wrong for your data. Removing identifiers is the bluntest option and the easiest to regret, so leave it last.

Does Macie work across accounts in an organization?

Yes, through AWS Organizations with a delegated administrator. Design around this: the administrator’s automated discovery settings apply to member accounts, members can’t change them, and members see coverage and sensitivity data for their own buckets but not the sensitive data findings themselves. That shapes who can actually remediate what.

The one thing worth remembering

Amazon Macie PII detection is only as good as its coverage, and coverage fails silently by design. An object with the wrong extension, the wrong storage class, or an unreachable encryption key produces no finding, and no finding renders identically to a clean result. Before you hand a Macie report to anyone who will make a decision from it, put the coverage numbers next to it: how many objects were classifiable, how many were analyzed, how many were skipped and why.

Do that once and the rest of the service becomes straightforward. Skip it and you’re maintaining an expensive dashboard reporting on whichever subset of your data happened to be readable.

Need help getting Macie to actually see your data?

Most of the work in a Macie rollout isn’t turning it on. It’s the unglamorous part: proving what got scanned, fixing the reasons things didn’t, and making the output land somewhere a human acts on it. That’s the part I help with.

• Coverage audits across your S3 estate, with per-bucket classifiable, analyzed and skipped counts and the specific cause of each gap
• Fixing those causes: KMS key policies, bucket policies blocking the service-linked role, lifecycle rules archiving data out of scope, pipelines writing extensionless objects
• Custom data identifiers and allow lists tuned against your real data and tested on positive and negative samples first
• Cost shaping: working out whether object monitoring or data inspection drives your bill, and scoping discovery so the number is defensible
• Findings pipelines through EventBridge or Security Hub into the alerting and ticketing you already run, with severity filtering that keeps signal above noise
• Multi-account setup under AWS Organizations, including the delegated administrator model and who can see and remediate what

If you want a second opinion, send me the output of the describe-buckets query above, or a screenshot of your coverage page, and I’ll tell you what’s actually being scanned.

The post Amazon Macie PII Detection: The Buckets It Never Opened appeared first on John Nessime.

Nobody looks at the old link. If it was signed with a long expiry and mailed out, it is still live: in the client’s inbox, in the inbox of the bookkeeper on the CC line, and in whatever mail archive that firm runs. It keeps working for anyone holding the URL until the clock runs out, because a presigned S3 URL is a bearer token. AWS says exactly that in its own documentation. The URL does not know who you are. It knows that you have it.

That is the shape of most document vault failures at a small accounting firm. Not a wide-open bucket. Not a stolen partner password. An ordinary delivery mechanism that leaks quietly and writes no log line that looks wrong six months later.

This post covers how to build a secure client document vault for an accounting practice: what the regulation actually demands, the six failure families that decide whether it holds, and the control that closes each one. The examples are AWS because that is where most of these land, but the failure families are platform independent.

Let the regulation write your requirements

Most firms treat compliance as paperwork done after the build. That is backwards. For a US accounting or tax practice, the FTC Safeguards Rule at 16 CFR Part 314 is the closest thing you will get to a free requirements document.

It treats tax preparers, bookkeepers and accounting firms as financial institutions under the Gramm-Leach-Bliley Act. Size does not get you out of it. The parts that translate directly into architecture:

• Access controls limiting each user to the customer information they need, and limiting customers to their own information.
• Encryption in transit over external networks and at rest, with compensating controls only where encryption is infeasible and your Qualified Individual signs off in writing.
• Multi-factor authentication for any individual accessing any information system, subject to the same written-approval exception.
• Secure disposal, with an outer bound of two years after last use unless retention is required by law or necessary for legitimate business purposes.
• Logging of authorized user activity, plus a named Qualified Individual who owns the program.

One more clause should shape your design more than any other. The Rule requires notifying the FTC within thirty days of discovering a security event involving the unencrypted information of at least five hundred consumers, and it counts encrypted information as unencrypted if the key was accessed by an unauthorized person.

Read that twice. Encryption only helps at the breach-analysis stage if the key was genuinely out of reach. That is the entire argument for treating key custody as a control separate from storage.

Separately, IRS Publication 4557 expects paid preparers to keep a Written Information Security Plan, and PTIN renewal asks you to confirm you have one. Treat the WISP as the document describing the vault you built, not a substitute for building it.

Failure one: the link that outlives the engagement

Presigned URLs are the sensible answer to “let the client download this without an AWS account”. The problem is their expiry semantics, which almost nobody has read.

Two clocks apply and the shorter wins: the expiry you asked for, and the lifetime of the credential you signed with. Long-lived IAM user credentials under Signature Version 4 reach seven days. A role session from AssumeRole dies when the session ends, one hour by default, regardless of what you requested. An EC2 instance profile is bound by metadata credential rotation. The console caps at twelve hours.

So the two symptoms are opposites and both are common. Links dying early is a credential problem. Links living too long is a policy problem, and that is the one that hurts.

The fix is not a smaller number. It is changing what goes in the email.

1. Email a notification, not the file link. The client logs in with MFA and the portal mints a URL at click time.
2. Keep that expiry in minutes. The URL only has to survive a redirect and a download.
3. Scope every URL to one object and one verb.
4. If you truly need a long-lived share, use CloudFront signed URLs or signed cookies, where you can revoke by rotating the key group without touching the object.

# Sign a download URL valid for five minutes.
# --expires-in takes seconds; the default is 3600.
aws s3 presign s3://firm-client-vault/clients/acme-llc/workpapers.pdf 
    --expires-in 300

Run that under an assumed role, wait past the session expiry, and watch it fail well before five minutes. That is behaviour worth seeing once in a test rather than discovering during an incident review.

Failure two: everyone inside the firm sees everything

This shows up in nearly every small-firm build, and the reasoning is always the same. Nine people, everyone helps with everything during filing season, and compartmentalising access is friction exactly when friction is expensive.

The case for flat access is real, so say it out loud: a scoped model people fight gets worked around, and a workaround is worse than the flat model you started with. If your access control makes a senior accountant email a PDF to themselves at 11pm, you have made things less safe.

So scope by engagement rather than by person, and automate the grant so nobody files a request. Make the key prefix the boundary, then grant the prefix instead of the bucket. Two statements, two jobs: the first controls what someone can enumerate, the second what they can fetch.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOnlyThisClient",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::firm-client-vault",
      "Condition": {
        "StringLike": { "s3:prefix": "clients/acme-llc/*" }
      }
    },
    {
      "Sid": "ReadWriteThisClient",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::firm-client-vault/clients/acme-llc/*"
    }
  ]
}

Without the prefix condition on ListBucket, a scoped user still gets a full inventory of your client list. Client names are confidential in this business. A directory listing that reads like a book of business is a disclosure even when no file opens.

The other half of this family is standing administrative access. Somebody has to fix things, so somebody ends up with a permanent s3:* on the whole bucket. That principal is the highest-value target in the firm and usually has the weakest MFA, because it is a service account nobody wants to touch.

Replace it with an assumable break-glass role: MFA required, short session, and an alert on every assumption. The alert is the point. Emergency access used twice a year is fine. Used weekly, it means your normal path is broken.

Failure three: encrypted at rest, key sitting right next to it

Default bucket encryption gets ticked in week one and forgotten. It satisfies the letter of “encrypted at rest” and protects against exactly one threat: somebody walking out of a data centre with a disk. That is not your threat.

Go back to the notification clause. If an attacker reaches a principal that can both read the object and use the key, your data is unencrypted for reporting purposes. The question is not “was the bucket encrypted”. It is “could whoever got in also use the key”.

• Use a customer managed KMS key. Only that gives you a key policy you control.
• Name the principals allowed to call Decrypt, and keep that list shorter than the list allowed to call GetObject. Two different lists means an attacker needs two wins.
• Use encryption context so grants can be conditioned per client, and so CloudTrail entries are readable later.
• Enable S3 Bucket Keys to cut KMS request volume. KMS bills per API call, so a vault serving many small reads generates a surprising line item.

Encryption in transit needs its own control, because IAM policies say nothing about the connection. A bucket policy does:

{
  "Sid": "DenyPlaintextTransport",
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": [
    "arn:aws:s3:::firm-client-vault",
    "arn:aws:s3:::firm-client-vault/*"
  ],
  "Condition": {
    "Bool": { "aws:SecureTransport": "false" }
  }
}

An explicit deny beats every allow, including one you add by mistake later. Five lines that permanently remove a category of misconfiguration, which makes it the best value item on this page.

Failure four: retention rules that fight your disposal duty

Here is the trap, and it is why I would not let an accounting firm switch on Object Lock without a conversation first.

Object Lock has two retention modes. In governance mode, a principal holding s3:BypassGovernanceRetention can remove or shorten the lock. In compliance mode, nobody can. Not an administrator, not the account root user. AWS is explicit that the only way to delete a compliance-locked object before its retention date is to close the AWS account.

Set that against the Safeguards Rule’s two-year outer bound on disposal of information you no longer need. Apply a blanket seven-year compliance-mode retention to everything landing in the vault and you have written a promise you cannot revoke, on documents you may later be obliged to destroy.

1. Classify by document type before client. Filed returns, engagement letters and supporting workpapers have real statutory retention arguments. Raw intake uploads and scratch scans usually do not.
2. Default to governance mode. It stops accidental and casual deletion, which is what you need, while keeping a documented and audited escape hatch.
3. Give s3:BypassGovernanceRetention to almost nobody and alert on every use through CloudTrail.
4. Reserve compliance mode for records where an external mandate genuinely requires immutability and you chose the period deliberately.
5. Use legal hold, not a retention period, for litigation. A legal hold has no end date and stays on until explicitly removed, which is right when you cannot know how long a matter runs.

Two details catch people out. Object Lock requires versioning and normally has to be enabled at bucket creation, so it is a design decision rather than a retrofit. And lifecycle expiration will not delete a version still under retention, so lifecycle and Object Lock are two layers that must agree, not alternatives.

# What lock configuration is this bucket actually running?
aws s3api get-object-lock-configuration --bucket firm-client-vault

# Freeze one object version for a matter with no known end date.
aws s3api put-object-legal-hold 
    --bucket firm-client-vault 
    --key clients/acme-llc/2023/return.pdf 
    --legal-hold Status=ON

For deletion itself, crypto-shredding is worth knowing. If a client’s documents sit under a per-client KMS key, scheduling that key for deletion renders every object unreadable without touching the objects. Clean, fast, provable, and irreversible once the waiting period ends, so it belongs behind two-person approval rather than in a script.

Failure five: no evidence when somebody asks

Every control above is worth nothing if you cannot show it held. The Rule expects logging of authorized user activity and an annual written report from the Qualified Individual.

The trap is that CloudTrail does not record S3 object-level reads and writes by default. Management events cover things like changing a bucket policy. Data events cover the actual GetObject calls, and they are opt-in and billed per event. Firms enable CloudTrail, see a trail in the console, assume they have a record of file access, then find during an incident that they have nothing at object level.

Turn on data events for the vault bucket. At firm scale the volume is manageable, and it is the difference between “we believe two people opened that file” and being able to show it.

Then decide where logs live. Access records for client financial data are themselves sensitive, so keeping them in the same account under the same administrators defeats the purpose. A separate logging account, or at minimum a separate bucket with its own key and retention, is the shape you want. If you already run Grafana Cloud or similar for infrastructure monitoring, pointing vault access logs there gives you dashboards and alerting without standing anything new up.

The alerts worth having are boring and few: any use of the break-glass role, any change to the bucket or key policy, any bulk download outside normal working patterns, and any failed decrypt. Four alerts that fire almost never beat forty that everyone mutes.

Failure six: the two ends of the pipeline

A vault only protects documents that are in it. The two ends are where firms quietly undo everything.

Intake

If clients can still email a W-2 as an attachment, they will, because it is easier. Each attachment becomes customer information sitting in a mailbox, a sent-items folder and a mail archive, each with a different retention policy and access model than your vault.

A policy memo will not fix this. Make the vault path faster than the email path: an upload link that works on a phone, no account creation for a one-off drop, and a mail rule that returns certain attachment types with the upload link instead. Then sweep the mailboxes you already have, because that backlog is real customer information and it counts.

Offboarding

Seasonal staff are normal here, so offboarding happens in batches and gets rushed. Disabling the SSO login is the part everyone does. The parts that get missed:

• Active role sessions and refresh tokens, which survive the account being disabled until they expire on their own.
• Presigned URLs that person generated, which stay valid because they are bound to the signing credential, not the login session.
• Named principals still sitting in the KMS key policy.
• Local copies on laptops and external drives. When hardware is retired, a certified erase tool such as O&O SafeErase is the difference between disposal and hoping.

The same applies to service providers. The Rule makes you responsible for selecting providers capable of maintaining appropriate safeguards and holding them to it contractually. If an outsourced bookkeeping team touches your document workflow, that relationship is in scope, and direct network access means they need MFA too. Cloudflare Zero Trust or a business VPN such as NordLayer in front of the portal gives you somewhere to enforce that without rebuilding the application.

Troubleshooting a secure client document vault

• Links expire far earlier than the expiry you set. You signed with temporary credentials that expired first. Sign from a principal whose credential outlives the link, or shorten the link and mint on demand.
• Uploads fail with access denied despite an allow on PutObject. With a customer managed key, both the IAM policy and the key policy must permit the caller. Check kms:GenerateDataKey, not just S3 actions.
• A file will not delete and no policy seems to block it. Look for Object Lock retention or a legal hold on that version. They are independent and either one alone blocks the delete.
• CloudTrail shows the bucket being created but no file access. Data events are not enabled. Management events will never show GetObject.
• A scoped user can see every client folder name. The prefix condition is missing from ListBucket, or was applied to the object ARN instead of the bucket ARN.
• Lifecycle rules appear to do nothing. Versions under retention are not expired by lifecycle. Confirm lock state before assuming the rule is wrong.

Common mistakes, and what to do instead

• Emailing the download link. Email a notification that a document is waiting, and mint the link after the client authenticates.
• Treating default bucket encryption as finished. Use a customer managed key and keep the decrypt list shorter than the read list.
• Granting bucket-wide list permissions. Condition ListBucket on the client prefix, or the roster leaks even when no document opens.
• Applying compliance-mode Object Lock broadly because it sounds safer. Write the retention schedule per document class first, get partners to sign it, then configure to match.
• Keeping a standing full-access principal. Replace it with an MFA-gated break-glass role that alerts on every assumption.
• Building a portal and leaving email intake open. Both channels then carry client data and only one is governed. Close the mail path and sweep the backlog.
• Storing access logs beside the data they describe. Ship S3 data events somewhere with different administrators and different keys.
• Never testing any of it. Verify a restore and a scoped-access denial quarterly. An unverified control is an assumption.

Frequently asked questions

Does a small accounting firm really fall under the FTC Safeguards Rule?

Almost certainly, if it prepares returns or handles client financial data in the United States. The Rule’s definition of a financial institution covers activities financial in nature, and the FTC’s own examples include tax preparation firms. A narrow exception at 16 CFR 314.6 covers institutions holding information on fewer than five thousand consumers, but it exempts only four specific provisions. Encryption, MFA, access controls and disposal still apply. Confirm your own position with counsel, not a blog post.

Should I build this or buy a client portal product?

Buy, if the product genuinely does what you need. A firm with no engineering capacity is safer on a maintained product than a bespoke stack nobody patches. Build when you need controls the product will not give you: per-client key isolation, your own retention logic, or integration with a pipeline you already run. The honest test is whether someone will still own the system in a year.

Can I self-host a secure client document vault instead of using AWS?

Yes. Nextcloud or similar on a well-run VPS from a provider such as InterServer or Contabo can meet the same requirements, and it is often simpler to reason about. What you take on is patching, backups, TLS renewal and log retention. The regulation does not care where the data sits. It cares that the controls exist and that you can show they held.

How long should client documents be retained?

There is no single number, which is why a schedule beats a default. Tax and professional standards drive minimums for filed returns and engagement documentation, and those vary by jurisdiction and engagement type. The Safeguards Rule pushes the other way. Write both directions down per document class, then configure to match.

Is a presigned URL safe to send to a client?

It is safe to use, not safe to send. Treat it as a bearer token with a short life, minted after the client authenticates and consumed immediately. What travels over email should be a notification, not the credential.

Does encrypting the bucket mean I avoid breach notification?

Only if the key stayed out of the attacker’s hands. The Rule treats encrypted customer information as unencrypted where the key was accessed by an unauthorized person. That is the whole argument for keeping key permissions narrower than storage permissions.

What is the minimum viable version for a two-person practice?

One bucket with per-client prefixes, a customer managed key, an explicit deny on non-TLS transport, MFA on every human account, short mint-on-click links, S3 data events enabled, and a written retention schedule. That is a weekend of work and it closes five of the six failure families above.

The one thing worth remembering

A secure client document vault is not defined by the storage you picked. It is defined by how narrow the path is between an attacker and a readable file, and by your ability to prove afterwards how many people walked it.

Most firms get the storage right and lose at the edges: a link that outlives the engagement, a key policy as wide as the bucket policy, a retention setting that cannot be undone, an audit trail nobody turned on. Fix those four and the rest is ordinary engineering.

Where I can help

I work with small firms and the MSPs that support them on exactly this kind of build. Things I take on regularly:

• Designing and building the vault itself: bucket layout, per-client prefix isolation, KMS key policy, and the portal glue that mints short-lived links.
• Reviewing an existing setup against the Safeguards Rule’s technical elements and producing a gap list your Qualified Individual can work from.
• Untangling retention: turning a vague policy into a per-document-class schedule, implemented with lifecycle rules, Object Lock and legal holds that do not conflict.
• Standing up the evidence layer: S3 data events, a separate logging account, dashboards, and the small set of alerts worth waking up for.
• Closing the email intake path without making life harder for clients, including mobile upload flows and a sweep of the historic mailbox backlog.
• Offboarding and access reviews: break-glass roles, session revocation, key policy hygiene, and a repeatable quarterly check.

If you want a second opinion, send the thing itself: a bucket policy, an IAM policy, a CloudTrail event selector, or a description of how documents reach your clients today. It is much easier to say something useful about a real configuration than about a general question.

The post Building a Secure Client Document Vault Your Accounting Firm Can Defend appeared first on John Nessime.