That gap between what a table logically contains and what the object store physically holds is what Iceberg table maintenance exists to close. Iceberg never overwrites a file in place. Deletes and updates write new files and a new pointer, and the old files sit where they are until something explicitly removes them. If nothing does, the bill only moves one way.

This post covers where those bytes hide, why expiring snapshots often frees nothing, how orphan files accumulate without showing up in any metadata you would think to check, and the order to run cleanup so you do not corrupt a table while shrinking it.

Why deleting rows does not shrink anything

Every write produces a snapshot: an immutable view of the table at a point in time, pointing at a manifest list, which points at manifests, which point at data files. Nothing in that chain is mutated. A new commit builds a new chain that reuses most of the old files.

That design is what gives you time travel, rollback and safe concurrent writes. It is also why storage grows silently. A data file only becomes a deletion candidate once no retained snapshot references it. Delete a year of rows and you free nothing, because the snapshot from before the delete still points at every one of those files.

So maintenance is not an optimization. It is the mechanism by which deletes become real.

The four places your storage is actually sitting

When a lakehouse bucket balloons, the bytes are in one of four categories. Each needs a different tool, and confusing them is how teams end up running the same job for weeks with no effect.

1. Data files held alive by snapshots you never expired

This is the one everyone knows about and usually the largest share. Snapshots accumulate until something expires them, and nothing expires them by default.

Before changing anything, look at the table’s own metadata. Iceberg exposes it as queryable tables:

-- How many snapshots exist, and how far back do they go?
SELECT committed_at, snapshot_id, operation
FROM db.orders.snapshots
ORDER BY committed_at;

-- How much data does the CURRENT snapshot reference?
SELECT count(*) AS file_count,
       sum(file_size_in_bytes) AS bytes
FROM db.orders.files;

Compare that second number against what your object store reports for the table prefix. The difference is your maintenance debt. On a neglected table the referenced set is often a minority of the physical bytes.

Expiring snapshots is a Spark stored procedure:

CALL my_catalog.system.expire_snapshots(
  table => 'db.orders',
  older_than => TIMESTAMP '2000-01-01 00:00:00',
  retain_last => 10,
  stream_results => true
);

Three arguments matter. older_than sets the age cutoff. retain_last is the floor: it keeps that many ancestor snapshots regardless of age, which is your rollback safety net. stream_results streams the delete list back to the driver instead of collecting it at once, which on a table with long history is the difference between the job finishing and the driver running out of memory.

The trap is that Iceberg treats these conditions as guidance rather than a strict contract, leaning toward retention when they conflict. An aggressive older_than plus a generous retain_last gives you the generous answer. Safe, but it surprises people who expected the timestamp to win.

Branches and tags are separate: they are not subject to snapshot age at all. A tag created for a regulatory export a year ago pins its entire snapshot chain, and no amount of expiring will touch it. Reference age is governed by history.expire.max-ref-age-ms.

2. Orphan files that no metadata points at

This is the invisible one, and the reason many teams conclude that maintenance “does not work.”

When a Spark task writes a data file and the job then fails before the commit lands, that file is already on disk. It is in no manifest, no snapshot, no metadata file. Expiry will never find it, because expiry compares the file sets of expired snapshots against retained ones, and a file that was never in a snapshot is in neither set.

Streaming writers, speculative execution, retried stages and killed compaction jobs all produce these. The only way to find them is to list the storage prefix and diff it against the metadata:

-- Always dry run first. This lists candidates without deleting.
CALL my_catalog.system.remove_orphan_files(
  table => 'db.orders',
  dry_run => true
);

Read that output before you run it live. Recent timestamps or unfamiliar paths mean stop and investigate.

The safety mechanism is older_than, which defaults to three days. That default is not arbitrary caution. Orphan removal cannot distinguish a file abandoned by a dead job from one being written right now by a live one. Set the interval shorter than your longest-running write and the procedure can delete a file a job is about to commit, corrupting the table. Shorten it only when you know your write durations, and never below your longest backfill.

The second hazard is less obvious. Iceberg compares file paths as strings. If your metadata records s3a://bucket/... and your listing returns s3://bucket/..., or you migrated between HDFS namenodes and the authority changed, every live file looks like an orphan. Running that deletes the table. The procedure defends against it by erroring on a prefix mismatch by default, via prefix_mismatch_mode with ERROR, IGNORE and DELETE as options. Resolve a mismatch with equal_schemes or equal_authorities. Do not reach for DELETE to silence it.

3. Metadata JSON files and manifests

Every commit writes a new table metadata JSON file. On a batch table that is a handful a day. On a streaming table committing every minute it is a lot of small objects, and those cost you in request charges and planning time as much as in bytes.

ALTER TABLE db.orders SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled' = 'true',
  'write.metadata.previous-versions-max' = '20'
);

The first property is off by default. The second defaults to 100 and controls how many previous metadata files are tracked in the metadata log. The catch is that cleanup only deletes tracked files. If you have been running with deletion disabled, files that fell out of the log are already untracked, and turning the property on later will not reach back for them. They are orphans now, and only orphan file removal clears them.

You can see the tracked set with SELECT timestamp, file FROM db.orders.metadata_log_entries.

Manifests rarely dominate the bill, but a table with thousands of tiny ones plans queries slowly. rewrite_manifests consolidates them. Treat it as a query performance job and run it after the deletion work, so it is not reorganizing metadata about to be thrown away.

4. The object store’s own hidden inventory

Most Iceberg guides skip this category, and it often explains a bill nobody can account for. Iceberg has no visibility into any of it.

• Incomplete multipart uploads. Large Parquet files upload in parts. When a writer dies partway, the completed parts sit in the bucket, billed as storage, and they do not appear in a normal object listing. Nothing removes them automatically.
• Noncurrent object versions. With versioning enabled, every file your maintenance job “deletes” becomes a noncurrent version you keep paying for, indefinitely, unless a rule expires it.
• Delete markers. Deletes on a versioned bucket leave markers behind. Small individually, but on a table with millions of expired files they add up and slow listings down.

A lifecycle configuration covers all three. On S3 the shape looks like this:

{
  "Rules": [
    {
      "ID": "abort-stale-multipart",
      "Status": "Enabled",
      "Filter": { "Prefix": "warehouse/" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    },
    {
      "ID": "expire-noncurrent-versions",
      "Status": "Enabled",
      "Filter": { "Prefix": "warehouse/" },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 14 },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    }
  ]
}

Now the warning that matters most here. Never put an age-based expiration or storage class transition rule on the prefix holding live Iceberg data files. A rule deleting objects older than ninety days will happily delete data files a current snapshot still references, and the table breaks with missing-file errors on the next read. Transitioning them to an archive class is just as bad, because reads then fail or stall on retrieval.

The rules above act only on things the table does not reference: abandoned upload parts, superseded versions, tombstones. Iceberg decides what live data to delete; lifecycle rules clean up after that decision.

This applies equally on S3-compatible storage. If you run a lakehouse on Cloudflare R2, Backblaze B2, Wasabi or Contabo Object Storage to dodge egress charges, check which lifecycle actions the provider actually implements. Support for aborting incomplete multipart uploads varies, and a provider that silently ignores the rule leaves you paying for parts you cannot see.

Running Iceberg table maintenance in a safe order

Order matters, and the reason is not stylistic. These procedures read each other’s inputs.

1. Compact first, if you are compacting. rewrite_data_files writes new files and leaves the old ones referenced by older snapshots. Doing it after expiry means the space you just freed is immediately replaced.
2. Expire snapshots. This is what turns unreferenced data files into deleted ones. It must happen before orphan removal, because expiry needs to read the manifests belonging to the snapshots it is dropping.
3. Remove orphan files. Only after expiry has committed. Running the two concurrently is a genuine race: orphan removal can delete files the in-flight expiry job still needs to read, and expiry fails.
4. Rewrite manifests, if planning is slow. Optional, and worth measuring before and after rather than running on faith.

On cadence: expiry is cheap and should run often, daily or better on high-commit tables. Orphan removal is expensive because it lists the entire table prefix, and the official guidance is to run it periodically but not frequently. Weekly on a busy table, monthly on a quiet one. Where supported, the prefix_listing option makes that scan substantially cheaper on object stores, where recursive directory listing is the slow part.

Letting table properties do the boring part

Procedures are what you run. Properties are what you set once so the procedures behave sensibly without a config file per table.

ALTER TABLE db.orders SET TBLPROPERTIES (
  'history.expire.max-snapshot-age-ms' = '604800000',   -- 7 days
  'history.expire.min-snapshots-to-keep' = '10'
);

The snapshot age default is five days and the minimum to keep defaults to one. Setting both means a generic maintenance job can call expire_snapshots with no tuning and each table gets its own policy. A regulated reporting table keeps ninety days; a staging table keeps an hour. That beats maintaining per-table retention in your scheduler.

The other property worth knowing is gc.enabled. Setting it false blocks physical deletion entirely. It exists for tables whose files are shared with something else, and it is correct for those. It is also a common accidental cause of “maintenance runs successfully and frees nothing.” Check it before debugging anything else.

Troubleshooting: the job succeeded and nothing shrank

• The counters were zero. expire_snapshots returns counts of deleted data files, delete files, manifests and manifest lists. All zero means nothing qualified. Check whether retain_last or the table’s minimum-to-keep property is holding everything.
• A branch or tag is pinning history. Refs do not age out with snapshots. Look at the table’s refs metadata table and see what is anchored.
• Garbage collection is off. Either gc.enabled is false, or the table was created by the snapshot procedure, which is barred from expiry precisely because it does not own its data files.
• Files were deleted but the bucket did not shrink. Versioning. The deletes created noncurrent versions. This is the most common cause of healthy counters and a flat bill.
• Orphan removal hit a file limit. Some managed implementations cap deletions per run. On a backlogged bucket you may need several passes before the number stops dropping.
• The driver died. On a table with very long history, add stream_results and raise max_concurrent_deletes so deletes spread across a thread pool instead of serializing.

Common mistakes

• Running orphan file removal with a short retention window on a table that has long-running writers. This is the one that can actually destroy data.
• Setting a bucket-wide expiration rule to “clean up the lake” and taking live data files with it.
• Enabling metadata deletion after the fact and expecting it to collect the already-untracked backlog.
• Scheduling expiry and orphan removal as two independent cron jobs that can overlap.
• Skipping the dry run because the last twenty were fine. The one that matters is the run after somebody changes the storage endpoint.
• Setting retention from a storage target rather than a recovery target. Ask how far back you need to roll back, then keep that much.

Best practices worth adopting

• Express retention as a table property, not a job argument. It travels with the table and survives whoever wrote the scheduler.
• Track referenced bytes against physical bytes per table as a metric, not an occasional query. The ratio drifting upward is the early signal; the invoice is the late one. Any time series backend you already run will do.
• Give maintenance its own compute and its own credentials. It is the only workload in the lake that needs delete permissions.
• Log the returned counters from every run. When someone asks in six months where the storage went, that log is the answer.

Frequently asked questions

How often should I expire Iceberg snapshots?

As often as your commit rate justifies. A streaming table committing every minute benefits from daily or hourly expiry; a nightly batch table is fine with weekly. Set the window from how far back you would realistically need to roll back, then keep a few extra snapshots as a floor.

Does expiring snapshots delete my data?

It deletes data files that no retained snapshot references. Current table contents are never affected. What you lose is the ability to time travel to or roll back to the expired snapshots, so agree the window with anyone who queries the table as of a date.

Why do orphan files exist if Iceberg commits are atomic?

The commit is atomic; the writes preceding it are not part of that atomicity. Data files are written first, then the commit makes them visible. If the job dies between those steps, the files exist and the commit never happened. Atomicity guarantees readers never see a partial table, not that failed writes clean up after themselves.

Can I use an S3 lifecycle rule instead of Iceberg maintenance?

No, and trying is how tables get broken. Lifecycle rules act on object age. Whether Iceberg still needs a file has nothing to do with age; a data file written two years ago can be referenced by the current snapshot. Lifecycle rules are right for upload parts, noncurrent versions and delete markers, and wrong for anything the table might still point at.

Do I need Spark to run Iceberg table maintenance?

Spark has the most complete set of procedures, but it is not the only route. Flink has its own maintenance API, Trino exposes expiry and orphan removal as table procedures, and several managed catalogs run scheduled maintenance for you. The mechanics are identical regardless of engine, because the work is defined by the table format rather than the compute. Only the syntax and the exposed arguments change.

Is orphan file removal safe to automate?

Yes, once you have verified that the retention window comfortably exceeds your longest write and that the paths in your metadata match those your storage listing returns. Automate it after watching a few dry runs, not before. Erroring on a prefix mismatch is a stop signal, not something to configure away.

The one thing to take away

Iceberg table maintenance is not housekeeping you get to postpone. It is the step where logical deletes become physical ones, and until it runs, storage only grows. The bill is the visible symptom; the same neglect also makes query planning slower and recovery windows fuzzier.

Set retention as a table property so the policy lives with the data. Expire snapshots on a schedule matching your commit rate. Remove orphan files less often, always after expiry, always with a window longer than your slowest write. Then add the storage lifecycle rules that clean up what Iceberg cannot see. Any one alone leaves money on the table.

Need help getting your lakehouse storage under control?

Most of the work here is not writing the procedures. It is working out which of the four categories your bytes are in, and building a maintenance path that will not break a table at three in the morning. That is the kind of thing I do:

• Audit an existing lake and produce a per-table breakdown of referenced bytes against physical bytes, so you know where the money is before touching anything
• Build a scheduled maintenance pipeline with correct sequencing, dry-run gates and per-table retention driven by table properties
• Set retention against real rollback and time travel requirements, gathered from the people who depend on them
• Design storage lifecycle rules that clear incomplete uploads, noncurrent versions and delete markers without touching live data files
• Diagnose maintenance jobs that report success and free nothing, including prefix mismatches, pinned refs and disabled garbage collection
• Add table health metrics and alerting so drift shows up as a graph rather than a surprise on the invoice

If you want a second opinion, send me the output of your snapshots metadata table, your current table properties, or the counters from a run that did not do what you expected. That is usually enough to tell you where the bytes are.

The post Iceberg Table Maintenance: Why Your Storage Bill Keeps Growing appeared first on John Nessime.

Moving the files is the solved part. Which bucket they land in, and what sits in front of that bucket, decides whether you spend almost nothing or spend more than the server you were trying to shrink.

Cloudflare R2 vs Amazon S3 usually gets reduced to one sentence: R2 does not charge for data transfer out, S3 does, therefore R2. That is wrong in both directions, and understanding why is most of the value here. This post covers the failure mode that shows up after the migration rather than during it, an honest profile of each service, the levers that actually move the number, and a procedure you can run against your own traffic in an afternoon.

Free egress is not free serving

R2 charges nothing to move bytes out, on any storage class. It still charges for the read. Every GetObject and every HeadObject is a Class B operation, and a media library made of thousands of small files racks those up fast. Cloudflare’s own asset hosting example in the R2 pricing docs makes the point: storage falls inside the free tier, writes fall inside the free tier, and the whole bill is read operations.

So the meter is not gigabytes. The meter is requests that reach the bucket, which means the variable deciding your bill on either platform is cache hit ratio.

This is where offloading setups go wrong without anyone noticing. The plugin rewrites image URLs to point at the bucket, images render, pages look fine, and every request for every thumbnail travels all the way to the origin and gets billed. Nothing breaks. Nothing alerts. The number just grows with traffic.

There is a sharper version on the R2 side. A new bucket gives you a public URL on an r2.dev subdomain, which is convenient and tempting. Cloudflare documents it as a testing endpoint with a variable rate limit. Push production traffic through it and requests get throttled with 429 Too Many Requests, throughput can be throttled too, and you get no cache, no WAF, no bot management. Those only exist once the bucket sits behind a custom domain you control. That is the invisible failure: not a broken image, a working setup billed on every request and rate limited under load.

Cloudflare R2: where it wins and where it hurts

Where R2 wins

• Egress really is zero. Not zero within a partner network, not zero up to a ratio of stored data. Zero, on both storage classes, through the S3 API and through Workers.
• A custom domain puts it behind the CDN. One managed CNAME and reads start being absorbed at the edge instead of hitting the bucket. No second product to buy and wire up.
• Deletes are free operations, along with aborting a multipart upload. Cleaning up a messy library costs nothing.
• Unauthorized requests are not billed. A caller without permission gets a 401 and you are not charged, which matters once someone starts hammering your bucket path.
• Migration tooling is free to use. Super Slurper copies a bucket across in bulk, Sippy migrates lazily on first miss. You pay only for the operations they perform against R2.

Where R2 hurts

• S3 compatible is not S3 identical. Cloudflare publishes a table of which operations are implemented and which are not. Real tooling has broken on the gaps: when AWS shipped SDKs that enabled CRC32 checksums by default, R2 rejected the header until the mismatch was resolved.
• There is no cold archive tier. Infrequent Access is as cold as it gets, and it carries a minimum storage duration, a retrieval fee, and doubled operation rates. For an active library that is the wrong tier, not a saving.
• Data lives in one primary location. Location hints are best effort rather than a region guarantee, and are honored only the first time a bucket with that name is created. Read performance away from that location comes from the edge cache, not from replicas.
• Plugin support is thinner. WP Offload Media, the long-standing WordPress option, officially lists Amazon S3, DigitalOcean Spaces, and Google Cloud Storage. R2 users generally land on Media Cloud, Advanced Media Offloader, Next3 Offload, or the S3-Uploads route driven from WP-CLI.

Amazon S3: where it wins and where it hurts

Where S3 wins

• Origin transfer to CloudFront is not billed. This quietly demolishes the usual comparison. AWS waives data transfer from an AWS origin to CloudFront, so a properly built S3 media stack never pays the direct-to-internet rate on cached traffic.
• The CloudFront free allowance is permanent, covering a monthly volume of data transfer out and requests rather than expiring after twelve months. Small sites can sit inside it indefinitely.
• Event driven derivatives are mature. Upload fires a Lambda that generates thumbnails, strips metadata, converts formats, writes the result back. Hard to beat if your media pipeline does real work.
• A full storage ladder with real lifecycle transitions, down to deep archive. If a large chunk of your library is genuinely cold, S3 has somewhere cheap to park it.
• Governance depth. Object Lock, versioning, replication, IAM condition keys, CloudTrail data events. When an auditor asks for a control, S3 usually has a named feature for it.

Where S3 hurts

• The default configuration is the expensive one. Point image URLs at the bucket endpoint and every byte leaves at the internet rate with nothing cached in front of it. The cheap path has to be built deliberately.
• Four meters instead of two: storage, bucket requests, CDN requests, CDN data transfer, plus per-feature charges. Forecasting takes real effort.
• Geography changes the rate. Delivery costs more per gigabyte to some regions than others, so an audience shift moves your bill without you changing anything.
• Leaving costs money. Copying a large library out is billed transfer, once, in a lump. Usually small against ongoing savings, but it is a real number somebody has to approve.

How to decide between Cloudflare R2 vs Amazon S3

Four levers move the outcome. Everything else is noise.

Cache hit ratio

The dominant term on both platforms, and the one people skip. At a high hit ratio the origin barely gets touched and both bills collapse toward the storage line. At a low hit ratio, R2 bills read operations and S3 bills origin GETs, and both climb with traffic. Media should be trivially cacheable because the files never change. If yours is not caching, that is a configuration problem worth fixing before you pick a vendor, because fixing it changes the answer.

Object count, not library size

WordPress does not store one file per upload. It stores the original plus every registered image size, and themes and page builders happily register more. Ten thousand uploads can easily be sixty thousand objects, each one a write on migration and a read on a cache miss. Count them before you model anything:

cd /path/to/wordpress

# Objects you are about to create in the bucket
find wp-content/uploads -type f | wc -l

# Bytes
du -sh wp-content/uploads

# How much of that count is derivative sizes rather than originals
find wp-content/uploads -type f -regextype posix-extended 
  -regex '.*-[0-9]+x[0-9]+.(jpe?g|png|webp|avif)$' | wc -l

That last number is usually the surprise. If most of your object count is thumbnails, trimming unused registered sizes before migrating is the cheapest optimization available, and it shrinks writes, reads, and storage at once.

How cold the library actually is

Most libraries are a long tail. A small set of recent files takes nearly all the traffic while the rest sits untouched for years. If that tail is large, S3 has somewhere genuinely cheap to put it and R2 does not. Be careful with R2’s Infrequent Access class here: it lowers the storage rate but raises both operation rates, adds a retrieval fee, and enforces a minimum duration whether you keep the object or not. It fits write-once, read-almost-never data, not a library anything still links to.

What the rest of your stack already is

If your application already runs in AWS and generates derivatives with Lambda on upload, moving to R2 means rebuilding that pipeline against Workers or R2 event notifications. That is real work you do not get paid for. If your site is a VPS from somewhere like InterServer or Contabo with Cloudflare already in front of it, R2 is nearly free work: the DNS is there, the cache is there, and the bucket slots in behind a subdomain you already control.

Setup details that change the answer

A few configuration choices matter more than the vendor choice. Get these wrong and the cheaper platform produces the bigger bill.

Serve from a custom domain, always

On R2 this is the difference between a cached, protected asset host and a throttled test endpoint. Attach one from the dashboard or from Wrangler:

npx wrangler r2 bucket domain add my-media-bucket 
  --domain=cdn.example.com 
  --zone-id=<YOUR_ZONE_ID>

npx wrangler r2 bucket domain list my-media-bucket

Then disable public access on the r2.dev subdomain, or you have left a second uncached, unprotected door into the same objects.

Verify the cache instead of assuming it

Assumptions about caching are where the money leaks. Ask the edge directly: request the same object twice and read the headers.

# Cloudflare in front of R2
curl -sI https://cdn.example.com/2024/07/photo-1024x768.jpg 
  | grep -iE 'cf-cache-status|cache-control|age'

# CloudFront in front of S3
curl -sI https://cdn.example.com/2024/07/photo-1024x768.jpg 
  | grep -iE 'x-cache|cache-control|age'

First request, expect a miss. Second, expect a hit. If the second still reports a miss, or reports a status meaning the response was never eligible for caching, every image view is costing you an origin read. Fix that before you compare anything.

Long cache lifetimes and versioned filenames

Uploaded media is immutable in practice. Nobody edits the bytes of photo-1024x768.jpg, they upload a new file. So cache lifetimes should be long and the object name should change when the content does. If you find yourself purging image caches regularly, the real problem is that your filenames are not versioned.

On the Cloudflare side, Smart Tiered Cache is worth enabling for R2 origins. It routes edge misses through an upper tier data center close to your bucket instead of letting every edge location fetch independently, which cuts the number of requests reaching R2 at all.

Clean up failed uploads

Large uploads use multipart. Interrupted ones leave orphaned parts that occupy billed storage and do not show up in a normal object listing. Both platforms support lifecycle rules for this, and on R2 you can manage them from Wrangler:

npx wrangler r2 bucket lifecycle list my-media-bucket

npx wrangler r2 bucket lifecycle add my-media-bucket 
  --name=expire-temp 
  --prefix=tmp/ 
  --expire-days=30

Configure an abort rule for incomplete multipart uploads on day one, whichever platform you land on. It is the most common source of storage you pay for and cannot see.

A decision procedure you can run this afternoon

1. Measure your current cache hit ratio for image paths, from analytics or a sampled read of access logs. Everything downstream depends on this number.
2. Count objects, not gigabytes. Run the find commands above and separate originals from derivative sizes.
3. Estimate monthly origin reads as total image requests multiplied by the miss rate. That, not your bandwidth, is what you feed into either pricing calculator.
4. Treat writes as a one-off migration spike plus a modest steady rate. Migration is usually the largest write event the bucket ever sees.
5. Price both against current published rates on the same day, using Cloudflare’s R2 calculator and the AWS pricing calculator. Rates move. Do not trust a number copied from a blog post, including this one.
6. Test with a prefix before committing. Offload one year of uploads, point the site at it, watch cache status headers and the operations dashboard for a week. A week of real traffic beats any spreadsheet.

If it comes out close, take R2. Not because it is cheaper in that scenario, but because it has fewer meters to reason about, and a simpler operational model is worth something on the day something breaks.

Arguments that don’t survive contact

“S3 charges egress, R2 doesn’t, so R2 always wins”

Only true against the worst possible S3 setup. Transfer from an S3 bucket to CloudFront is not billed, so a properly built S3 media stack pays CDN delivery rates on cache misses, not the direct-from-bucket rate. Add CloudFront’s permanent free allowance and a small site pays nothing on either platform. The gap opens at scale and on origin-heavy traffic, not on the first terabyte.

“R2 is a CDN”

R2 is object storage that can sit behind a CDN you already have. The caching, the WAF, the bot rules all come from the Cloudflare zone, and none of it applies to the development URL. If the bucket is not behind a custom domain on a zone you control, you have storage without delivery.

“S3 compatible means drop-in”

It means most tools work with a changed endpoint and a region value of auto. It does not mean every operation, header, and checksum behaves identically. Budget an afternoon for fighting an SDK default, and test uploads, multipart uploads, deletes, and signed URLs against the real bucket before cutting over.

“We’ll save money moving cold files to Infrequent Access”

Sometimes. The lower storage rate arrives with higher operation rates, a retrieval fee, and a minimum duration. If files are cold enough to justify that, they are probably cold enough to belong in a real archive tier, which R2 does not have. Model it against your read pattern rather than assuming a cheaper per-gigabyte number is a cheaper bill.

Frequently asked questions

Is Cloudflare R2 cheaper than S3 for a WordPress media library?

Usually, once traffic is meaningful and the library is served publicly. For a small site behind a well configured CDN, both can land near zero. R2’s advantage grows with egress volume and with the number of requests that miss cache, which is why measuring hit ratio comes first.

Can I use the same WordPress plugin for both?

Several plugins speak the S3 API and support both, including Media Cloud, Advanced Media Offloader, and Next3 Offload. WP Offload Media officially lists Amazon S3, DigitalOcean Spaces, and Google Cloud Storage, so check current provider support before assuming a swap is free. Whichever you pick, confirm it rewrites existing URLs and not only new uploads.

How do I migrate an existing S3 bucket to R2 without downtime?

Cloudflare offers two paths. Super Slurper copies everything in bulk. Sippy migrates incrementally, pulling an object from the source the first time it is requested and serving from R2 afterward. Both are free to use, you pay for the operations they perform against R2, and your source bucket may charge you for the reads.

Do I still need a CDN in front of R2?

You need a custom domain on a Cloudflare zone, which is what puts the CDN in front of it. That is not optional for production. The development URL is rate limited by design and gets no cache, WAF, or bot management.

Can I keep S3 and just put Cloudflare in front of it?

You can, and it does cut origin reads. Be aware that cache misses then pull from S3 across the public internet, billed as ordinary S3 egress, unlike an S3 origin sitting behind CloudFront. Reasonable as an interim step, rarely the cheapest end state.

What about Backblaze B2, Wasabi, or DigitalOcean Spaces?

All viable and worth pricing if you are already modelling. B2 competes on raw storage, Wasabi sells predictability, Spaces is convenient when your droplets are already there. The analysis transfers directly: find the meters, find your cache hit ratio, multiply.

How do I monitor the bill after migrating?

Watch operations and cache hit ratio, not storage. Storage grows slowly and predictably. Operations track traffic and configuration mistakes, which is where the unpleasant surprises live. A Grafana Cloud dashboard or a scheduled pull from the provider’s usage API is enough; the point is that somebody looks weekly rather than at invoice time.

Conclusion

If you take one thing from this Cloudflare R2 vs Amazon S3 comparison, take this: the vendor choice is second order. What decides your media offloading bill is how many requests reach the bucket, and that is a function of your cache configuration, your object count, and whether you put a real domain in front of the storage.

Fix the cache first. Count the objects. Then price both, on the same day, against your own numbers. Most of the time R2 comes out ahead for public media, S3 comes out ahead when the library is entangled with AWS services or needs a genuine archive tier, and the difference is smaller than the internet suggests.

Need help moving a media library without breaking it?

Media offloading looks like a plugin install and turns into a URL rewriting, caching, and permissions problem. I work with teams on the parts that are easy to get subtly wrong:

• Modelling R2 against S3 using your real object counts, request volumes, and cache hit ratio instead of a generic calculator
• Planning and running the migration: bulk or incremental copy, URL rewriting, and a rollback path if the cutover misbehaves
• Custom domain, cache rule, and header configuration so origin reads collapse instead of tracking your traffic
• Bucket permissions, CORS, signed URL flows for private downloads, and lifecycle rules for orphaned multipart uploads
• Trimming registered image sizes and derivative sprawl before migration so you stop paying to store thumbnails nothing links to
• Dashboards for operations, cache hit ratio, and storage growth so the bill stops being a monthly surprise

Send me a curl -I of one of your image URLs and a rough object count, and I can usually tell you quickly whether you have a pricing problem or a caching problem.

The post Cloudflare R2 vs Amazon S3 for Media Offloading: Where the Bill Actually Comes From appeared first on John Nessime.

That is where most cost work goes wrong. The pressure is on the total, so people attack the biggest line item first. The biggest line item is almost always compute, and compute is the one place where a wrong guess shows up as latency at peak. Three weeks later the bill is lower and the p99 is worse, and nobody connects the two, because the change that caused it was a Tuesday afternoon instance swap that “went fine.”

This post is about how to cut AWS costs without breaking production by ordering the work differently: not by how much each change saves, but by how far the damage travels if the change is wrong. I will go through the levers in that order, with the traps in each one, the commands I use to find the waste, and the handful of cuts that look harmless and are not.

The cuts that break production are rarely the ones you fear

The line items that look most alarming on a bill are usually networking and storage overhead: NAT Gateway processing, public IPv4 hours, log ingestion, snapshots nobody owns. They feel dangerous to touch because they sit in the middle of the VPC. In practice, almost all of them can be removed or rerouted with zero behavioural change to your application. A gateway endpoint for S3 does not alter a single byte your code sees. Deleting an Elastic IP associated with nothing cannot affect anything, by definition.

The changes that feel safe are the ones with real blast radius. Dropping an instance one size. Switching a volume type. Moving objects to a colder storage class. Each is a single API call, reversible on paper, and each quietly narrows a performance envelope you were relying on without knowing it. The failure does not show up at deploy time. It shows up the next time you hit the load level that used to fit inside the old headroom.

So the rule I work by: sequence cost work by blast radius, not by savings size. Take everything production cannot possibly notice first. Then the things that move a performance envelope, one workload at a time, with a metric to watch. Then, last, the financial commitments, which cannot break your uptime at all but can lock you into paying for architecture you are about to throw away.

First, find out what you are actually paying for

The default Cost Explorer view groups by service, which is close to useless here. “EC2 – Other” is not a thing you can act on. It is a bucket holding EBS volumes, snapshots, NAT Gateway hours, NAT Gateway data processing, data transfer and public IPv4 charges, all mixed together.

Group by usage type instead

Usage type names the actual mechanism being billed. Strings like NatGateway-Bytes, NatGateway-Hours, PublicIPv4:InUseAddress and EBS:VolumeUsage.gp2 tell you what to go fix. This pulls last month’s usage types sorted by cost:

aws ce get-cost-and-usage 
  --time-period Start=$(date -d '1 month ago' +%Y-%m-01),End=$(date +%Y-%m-01) 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=USAGE_TYPE 
  --output json 
  | jq -r '.ResultsByTime[0].Groups[] | [.Keys[0], .Metrics.UnblendedCost.Amount] | @tsv' 
  | sort -k2 -gr | head -30

That uses GNU date, so run it on Linux unless you have coreutils on macOS. Cost Explorer API calls are themselves billed per request, which surprises people the first time they wrap this in a cron job.

Billing data cannot tell you where your bytes went

This is the most useful thing to internalise about networking spend. Cost data knows how many gigabytes crossed the NAT Gateway. It has no idea whether they were headed to S3, to ECR, to CloudWatch or to a third-party API. There is no destination breakdown in billing data.

VPC Flow Logs are the missing half. Turn them on for the private subnets, then query them grouped by destination address with Athena or CloudWatch Logs Insights and map the ranges back to AWS services. Now you have the split: this much to S3, this much to ECR, this much genuinely to the internet. That is what tells you which endpoint is worth adding. Flow Logs cost money to ingest, so scope them tightly and turn them off once you have the answer. A week on the right subnets is enough.

Tier zero: savings production cannot notice

Everything in this tier changes billing without changing behaviour. Do all of it before you touch a single instance type.

The orphan sweep

AWS charges for allocated resources, not used ones. A volume attached to nothing bills at full rate. An Elastic IP attached to nothing bills by the hour, and has done ever since AWS began charging for public IPv4 addresses whether or not they are in use.

# Elastic IPs associated with nothing
aws ec2 describe-addresses 
  --query 'Addresses[?AssociationId==`null`].[PublicIp,AllocationId]' 
  --output text

# EBS volumes in the "available" state, meaning attached to no instance
aws ec2 describe-volumes --filters Name=status,Values=available 
  --query 'Volumes[].[VolumeId,Size,VolumeType,CreateTime]' 
  --output text

Run the same pass over load balancers with no healthy targets, snapshots whose source volume is gone, and old AMIs still holding snapshots behind them. Snapshot a volume before deleting it. The snapshot costs a fraction of the volume and buys you a rollback if something did care after all.

NAT Gateway is usually the largest risk-free win

A NAT Gateway bills three separate ways: an hourly charge for existing, a per-gigabyte charge for every byte it processes, and then whatever data transfer applies to where that traffic actually went. Teams look at the hourly rate, decide it is a rounding error, and never look at the processing line. The processing line is the one that grows with traffic.

The part that stings: traffic from a private subnet to S3 or DynamoDB in the same region is charged NAT processing even though it never leaves the AWS network. Gateway endpoints for those two services are free, with no hourly and no per-gigabyte charge. If your private subnets talk to S3 through a NAT Gateway, this is the best ratio of saving to risk available anywhere on the platform.

aws ec2 create-vpc-endpoint 
  --vpc-id vpc-0123456789abcdef0 
  --service-name com.amazonaws.eu-west-1.s3 
  --vpc-endpoint-type Gateway 
  --route-table-ids rtb-0123456789abcdef0

A gateway endpoint works by adding a prefix-list route to the route tables you name, so the main failure mode is naming the wrong tables and leaving traffic on the old path. Nothing breaks; you just do not save anything. The real caveat is policy: if you have bucket policies or endpoint policies that restrict access by source, check them first, because requests arriving through an endpoint present differently.

Interface endpoints for other services (ECR, CloudWatch Logs, Secrets Manager, SSM) are a different calculation. They charge per hour per availability zone plus per gigabyte, so they only pay off above a certain volume for that one service. Use the Flow Logs breakdown to decide. Pulling container images from ECR through NAT on every scale-out is the case that usually justifies one.

Log groups that never expire

CloudWatch Logs bills mostly on ingestion and secondarily on stored volume. A log group created without an explicit retention setting keeps data forever, so every debug group someone created during an incident years ago is still accruing storage.

# Log groups with no retention policy set
aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output text

aws logs put-retention-policy 
  --log-group-name /aws/lambda/my-function 
  --retention-in-days 30

Retention is the safe half. Reducing ingestion is a tier-one change, because you are removing evidence you might need during an incident. If you need a long retention window cheaply, exporting to S3 with a lifecycle rule beats paying CloudWatch storage rates indefinitely, and a log platform like Grafana Loki or a hosted equivalent gives you a cheaper long tail while CloudWatch keeps the recent window.

Incomplete multipart uploads

Failed large uploads leave parts behind in S3. Those parts are billed as storage and do not appear in a normal object listing, which is why some buckets bill for far more than they seem to contain. One lifecycle rule ends it:

{
  "Rules": [
    {
      "ID": "abort-incomplete-multipart",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

Apply it with aws s3api put-bucket-lifecycle-configuration. That call replaces the entire lifecycle configuration on the bucket, so fetch the existing one and merge rather than overwrite.

Tier one: savings that move a performance envelope

Now a wrong call has consequences. Everything here goes one workload at a time, with a named metric you watch for at least a full traffic cycle afterwards.

gp2 to gp3, and the throughput trap

This is presented everywhere as a free win, and for most volumes it is. gp3 costs less per gibibyte and gives every volume a baseline of 3,000 IOPS regardless of size, where gp2 scales at 3 IOPS per GiB and only reaches 3,000 sustained at 1 TiB. For a small database volume quietly burning through burst credits, gp3 is a straight upgrade in both cost and stability.

The trap is throughput, not IOPS. gp3’s baseline throughput is 125 MiB/s. A large gp2 volume can deliver up to 250 MiB/s. Migrate a big sequential-read volume to gp3 with the defaults and you halve its throughput ceiling while the cost report still shows a saving. Log processing, backup jobs and analytics scratch space are exactly the workloads that hit this, and exactly the ones where the symptom is “the nightly job got slower” rather than an alert.

Check VolumeReadBytes and VolumeWriteBytes in CloudWatch first, and provision throughput explicitly where the volume needs it:

aws ec2 modify-volume --volume-id vol-0123456789abcdef0 
  --volume-type gp3 --iops 3000 --throughput 250

aws ec2 describe-volumes-modifications --volume-id vol-0123456789abcdef0 
  --query 'VolumesModifications[].[ModificationState,Progress]' 
  --output text

Elastic Volumes changes the type in place with the instance running, so there is no downtime. Two things to watch anyway. If the root volume is declared inline in a CloudFormation block device mapping, changing its type there can replace the instance rather than modify the volume; modify through the EC2 API instead. And there is a cooldown before a volume can be modified again, so plan on getting it right rather than iterating. On RDS the same change also runs without downtime, but the instance then enters a storage optimization state during which further storage changes are blocked.

Rightsizing without guessing

AWS Compute Optimizer reads CloudWatch metrics and returns recommendations with a confidence rating. Better than eyeballing CPU graphs, with one important gap: without the CloudWatch agent installed it cannot see memory. A JVM service sitting at 12% CPU and 88% heap looks like an obvious downsize candidate and is not.

aws compute-optimizer get-ec2-instance-recommendations 
  --query 'instanceRecommendations[?finding==`OVER_PROVISIONED`].[instanceArn,currentInstanceType]' 
  --output text

My working rules:

• Look at a window that includes your worst week, not a rolling average. Averages hide the peak that justifies the size.
• Change one dimension at a time, family or size, so you know what caused any regression.
• Treat burstable families as a separate decision. A T-family instance that exhausts CPU credits under sustained load either throttles or silently bills you for unlimited mode. Neither is what you wanted.
• Moving to Graviton usually saves more than downsizing x86, but it is an architecture change. Images need to be multi-arch and every compiled dependency needs to build. That belongs in a sprint, not in a cost-cutting afternoon.
• Write down the metric you expect to move and the threshold at which you revert, before you make the change.

S3 storage classes and the small-object trap

Intelligent-Tiering is the low-thought option and it works well for what it was designed for: objects of meaningful size with access patterns you cannot predict. It charges a small per-object monitoring fee, moves objects between tiers automatically, and applies no retrieval fee on the automatic access tiers.

Where it fails is object count. Objects below the eligibility threshold of 128 KB are not auto-tiered at all. They sit in the frequent access tier permanently while still paying monitoring. A bucket holding hundreds of millions of thumbnails, metadata records or small JSON documents can pay for monitoring that produces no saving whatsoever. Check average object size first, from S3 Storage Lens or an inventory report: total bytes divided by object count.

Lifecycle transitions have their own trap, since each transition is a billed request. Moving a billion small objects to a colder class can cost more in transition requests than the storage saving returns in a year. Transitions pay off on size, not on count. Expiry is the exception: deleting costs nothing and removes the storage entirely. While you are in there, check non-current versions in versioned buckets, which accumulate invisibly and need their own rule. And if your workload is genuinely egress-heavy, an object store with no egress charge such as Cloudflare R2 changes the arithmetic enough to model, in exchange for cross-provider latency and an operational split.

Tier two: commitments, which break budgets rather than uptime

Savings Plans and Reserved Instances cannot cause an outage. They are billing constructs. What they can do is hold you to a spend rate for one or three years after the architecture that justified it has changed. What I would want anyone to know before signing:

• Savings Plans cannot be cancelled during the term; AWS documentation is explicit. There is a limited return path with eligibility conditions and quotas per billing family, but plan as though there is no exit.
• Standard Reserved Instances can be listed on the Reserved Instance Marketplace, which is a real exit but not a guaranteed one, since somebody has to buy. Convertible RIs cannot be sold, only exchanged for a different configuration.
• Commit after rightsizing, never before. A plan applied to an oversized fleet locks in the waste and removes the incentive to fix it.
• Savings Plans do not reserve capacity. If you need a guarantee that instances will be available in a specific zone, that is an On-Demand Capacity Reservation, a separate mechanism the discount can then apply to.
• Savings Plans do not apply to Spot usage or to usage already covered by an RI. If much of your fleet is Spot, your committable baseline is smaller than the bill suggests.
• Compute Savings Plans survive architectural drift: family changes, size changes and moves between EC2, Fargate and Lambda stay covered. EC2 Instance Savings Plans and Standard RIs discount more deeply and tolerate drift poorly. Choose based on how stable your architecture actually is, not how stable you would like it to be.

Most teams land on a layered shape: commit at the baseline you are confident will still exist in a year, leave the variable top of the curve on demand or on Spot. Flat usage and a frozen architecture justify deeper, longer commitments. Mid-migration, mid-container-adoption or mid-anything, take the smaller discount for the shorter term.

One honest question worth asking here: does the workload need to be on AWS at all? Steady, non-elastic things like build runners, staging environments and internal tooling are often cheaper on a plain VPS from a provider such as Contabo or InterServer, while genuinely elastic workloads are often cheaper on AWS. The trade is operational, because you take back patching, backups and capacity planning that AWS was doing for you.

Troubleshooting: the bill went down and something broke

The hard part of diagnosing a cost regression is the delay. The change lands on Tuesday, the symptom appears at month-end batch time, and by then three other things have shipped. Work backwards from the symptom:

• Latency got worse but CPU is fine. Look at storage. VolumeQueueLength climbing with throughput flat at a round number means you are sitting on a provisioned ceiling, which is the gp3 default-throughput signature. On gp2, a falling BurstBalance is the tell, and it usually starts hours before anything is user-visible.
• Intermittent timeouts after rightsizing. Check whether the new instance type has lower network or EBS bandwidth, not just fewer vCPUs. Baseline bandwidth scales with size, and smaller sizes burst their network the way T instances burst CPU.
• Something stopped reaching a service. If you added VPC endpoints, check whether an endpoint policy is denying it. Interface endpoints also need the endpoint’s own security group to allow inbound from your workload, which is the most common reason a new endpoint appears to break connectivity.
• Objects are slow or missing from an application path. Check whether a lifecycle rule moved them into a class with retrieval latency. The Glacier tiers inside Intelligent-Tiering are opt-in for exactly this reason, and enabling them turns instant reads into restore requests.

Keep a change log for cost work specifically: a dated line per change, what you expected to save, what metric you were watching. Five minutes each, and it is the difference between diagnosing this in an hour and in a week.

Common mistakes

• Starting with the biggest number. Compute is the biggest line and the riskiest to touch. It is the last tier, not the first.
• Committing before rightsizing. This locks in the exact waste you were trying to remove, for years.
• Accepting gp3 defaults on high-throughput volumes. Cheaper storage, halved throughput ceiling, no alert.
• Enabling Intelligent-Tiering on buckets full of tiny objects. Monitoring fees on objects that will never move a tier.
• Turning off monitoring to save on monitoring. Understandable at 2am on a budget deadline, indefensible during the next incident. Cut cardinality and retention, not coverage.
• Doing it once. Cost drifts back within two quarters unless something structural stops it.

Best practices that make the savings stick

• Make cost visible before it is a crisis. AWS Budgets with alerts at a percentage of forecast, plus Cost Anomaly Detection, catch the runaway job on day two rather than on the invoice. Platforms like Vantage, CloudZero or Kubecost for per-namespace Kubernetes attribution earn their place once several teams share an account structure.
• Tag at creation, in the IaC. Tags applied afterwards do not backfill historical cost data. If it was not tagged when it was created, that spend is unattributable forever.
• Put non-production on a schedule. Dev and staging running nights and weekends is the most common waste in any account, and stopping them has no production blast radius at all.
• Make the cheap thing the default in the module. gp3, sensible log retention, gateway endpoints and lifecycle rules belong in your Terraform or OpenTofu modules, so the next environment is born correct rather than remediated later.
• Review commitment coverage on a schedule. Coverage and utilisation are different numbers and both matter. High coverage with low utilisation means you over-committed.

Frequently asked questions

What is the fastest way to cut AWS costs without breaking production?

Add gateway endpoints for S3 and DynamoDB, delete unassociated Elastic IPs and unattached EBS volumes, and set retention on log groups that have none. All four are billing-only changes with no behavioural effect on your application, and together they often account for a large share of the surprise on a bill.

Why is my “EC2 – Other” line item so large?

It is a mixed bucket: EBS volumes and snapshots, NAT Gateway hours and data processing, data transfer, and public IPv4 address charges. Regroup Cost Explorer by usage type rather than by service to see which of those is driving it.

Is migrating from gp2 to gp3 always cheaper?

No. gp3 has a lower per-gibibyte price, but IOPS above the included baseline and throughput above the included baseline are billed separately. A volume needing high provisioned throughput can cost more on gp3 than it did on gp2. Model the volumes that genuinely need extra performance instead of assuming a flat saving across the fleet.

Can I cancel a Savings Plan if my usage drops?

Assume no. AWS documents Savings Plans as non-cancellable for the term. There is a limited return mechanism with eligibility rules and quotas, and no secondary marketplace of the kind that exists for Standard Reserved Instances. Size the commitment to the baseline you are confident about, not the peak you hope for.

Should I use Spot Instances to cut costs?

For fault-tolerant, interruptible work such as CI runners, batch jobs and stateless workers behind a queue, the discount is hard to ignore. For anything holding state in memory or serving a request that cannot be retried, the interruption notice is short and the failure is abrupt. Spot is an architecture decision, not a billing toggle.

How do I find out where my NAT Gateway traffic is going?

Billing data cannot tell you. Enable VPC Flow Logs on the relevant subnets, query them with Athena or CloudWatch Logs Insights grouped by destination address, and map the results back to AWS service ranges. That gives you the split between S3, ECR, other AWS services and the genuine internet, which is what determines whether an endpoint pays for itself.

The one thing worth remembering

If you take one idea from this, take the ordering. You cut AWS costs without breaking production by sorting the work by blast radius rather than by savings size, and by accepting that the frightening-looking networking and storage line items are usually the safe ones, while the innocuous single-API-call changes are the ones that bite.

Do the tier-zero sweep first and completely. It is free, it is fast, and it buys you the standing to say “not yet” when someone asks you to shrink a production database on a Thursday. Then move one performance envelope at a time, with a metric and a revert threshold. Commit last, at the baseline you actually believe in.

Need a second pair of eyes on your AWS bill?

This is work I do regularly, and it goes faster with someone who has no attachment to how the account got this way. Things I can help with:

• A usage-type level breakdown of your bill, with savings ranked by blast radius, so you know what is safe this week and what needs a change window.
• NAT Gateway and data transfer analysis from VPC Flow Logs, ending in a specific list of which endpoints to add and what each is worth.
• EBS and S3 review: gp3 candidates with throughput checked properly, lifecycle rules, version cleanup, and storage class decisions based on your real object size distribution.
• CloudWatch and logging spend reduction that keeps the signal you need during incidents, including moving long-tail retention off CloudWatch storage rates.
• Savings Plan and Reserved Instance modelling against your actual baseline, including what happens to coverage if the migration you are planning goes ahead.
• Guardrails so it stays fixed: Budgets, anomaly alerting, tagging enforced in Terraform or OpenTofu, and cheap defaults baked into your modules.

If you want a concrete starting point, send me a Cost Explorer export grouped by usage type, or the output of the orphan sweep commands above, and I will tell you what I would touch first and what I would leave alone.

The post Cut AWS Costs Without Breaking Production: A Blast-Radius Playbook appeared first on John Nessime.

Sometimes the disk really is dying. More often it isn’t, and the number you’re staring at is not measuring what you think it’s measuring.

This post covers troubleshooting high I/O wait the way I’d actually do it on a production box: what the metric means, why it lies in both directions, and how to work through the five failure families that produce it.

What high I/O wait actually measures

Here’s the definition that matters: iowait is a subcategory of idle time. For a given CPU, it’s the time that CPU spent doing nothing while at least one task scheduled on it had an outstanding block I/O request.

Two consequences fall out of that, and both will burn you. High iowait can mean nothing is wrong. If the box has spare CPU and one process is pulling a large file off disk, the CPU genuinely has nothing else to do. Copying a backup will push iowait toward 100 percent on an idle host, and that’s a working system, not a fault.

Low iowait can hide a severe problem. This is the one that costs people days. If the CPU has other runnable work, the kernel context switches to it and that time gets counted as user or system, not iowait. A busy application server can be stalling badly on storage while wa reads two percent, because the CPU is never actually idle.

Add the averaging problem on top. The wa figure in top is averaged across every core. One thread pinned to one core and blocked hard on disk shows up as roughly three percent on a 32-core machine. Press 1 in top to break out per-CPU lines before you conclude anything from the aggregate.

So iowait is a hint, not a verdict. Treat it as the thing that starts the investigation, never the thing that ends it.

Start with pressure, not percentages

Pressure Stall Information is the metric I reach for first, and it exists precisely because iowait is a poor proxy for pain. It answers the question you actually care about: how much time did tasks lose because they couldn’t get I/O?

cat /proc/pressure/io

You get two lines. some is the share of time at least one task was stalled waiting on I/O. full is the share of time every non-idle task was stalled at once, which is CPU capacity being burned for nothing. The avg10, avg60 and avg300 fields are percentages over the last 10, 60 and 300 seconds. total is a cumulative microsecond counter since boot, which is what you want for graphing because averages smooth away short spikes.

The practical rule I use: a rising full line on io is a real user-visible problem almost every time. A high some line with a flat full line usually means one noisy job is waiting while everything else gets on with its life.

If /proc/pressure doesn’t exist, the kernel was built without CONFIG_PSI, or it was built with PSI compiled in but disabled by default, in which case you need psi=1 on the kernel command line. On cgroup v2 systems you also get per-cgroup pressure, which is how you narrow from “the host is stalling” to “this container is stalling” in one command:

# Pressure for one systemd slice, useful on any modern distro
cat /sys/fs/cgroup/system.slice/io.pressure

# Pressure for every leaf cgroup that has it, sorted by nothing,
# just eyeball the full lines
find /sys/fs/cgroup -name io.pressure -exec sh -c 
  'printf "%s: " "$1"; sed -n 2p "$1"' _ {} ;

If you run Prometheus, node_exporter has exposed these since it gained a pressure collector. The counters are node_pressure_io_waiting_seconds_total for some and node_pressure_io_stalled_seconds_total for full. Rate those and you have a far better storage alert than any threshold on %iowait will ever be. Grafana Cloud and Netdata both chart PSI out of the box if you’d rather not build the panels yourself.

Failure family one: the device really is saturated

This is the case everyone assumes, so get it confirmed or eliminated quickly.

iostat -xy 1

The -x gives extended per-device stats. The -y matters more than people realise: without it, the first report is a since-boot average, and reading that first block is the single most common iostat mistake there is. It will tell you the disk has been fine all year, which is true and useless.

Now the columns worth your attention:

• r_await and w_await: average milliseconds a read or write took, queue time included. This is the closest thing to “how slow does storage feel”. Split reads from writes, because they fail very differently.
• aqu-sz: average queue depth. Requests either queued or in service. A depth that climbs while throughput stays flat is the signature of a genuinely saturated device.
• r/s, w/s, rkB/s, wkB/s: the actual workload. Compare against what the device or volume class is rated for.
• %util: treat with suspicion, see below.

The %util trap. That column counts the fraction of time at least one request was in flight. On a spinning disk that serviced one request at a time, it was a fair saturation signal. On NVMe and on cloud volumes that handle many requests in parallel, a device can sit at 100 percent while running at a fraction of its real capacity. Provisioning a bigger volume off the back of a 99 percent %util reading is a common and expensive way to change nothing. Use aqu-sz and the await columns instead. Similarly, svctm was deprecated and dropped from newer sysstat output because it was unreliable on parallel devices; ignore it if an old build still prints it.

Once you know the device is busy, find the culprit. pidstat -d 1 gives per-process read and write rates from the same sysstat package and needs nothing special enabled. iotop -o is friendlier, but on kernels from 5.14 onward per-task delay accounting is off by default, so the SWAPIN and IO> columns come back empty until you turn it on:

sudo sysctl kernel.task_delayacct=1
sudo iotop -oPa
# turn it back off when you're done, it costs a little scheduler overhead
sudo sysctl kernel.task_delayacct=0

And check the boring things before the clever ones. dmesg -T | grep -iE 'i/o error|reset|medium' catches a failing drive or a flapping controller. smartctl -a /dev/sda catches reallocated sectors on hardware you own. If either of those turns something up, stop reading and replace the disk.

Failure family two: memory pressure in an I/O costume

This is the failure family I’d check second, and it’s invisible if you only look at storage.

When RAM gets tight, the kernel starts evicting page cache and swapping. Both generate disk traffic. The disk gets busy, iowait climbs, and every graph points at storage while the actual constraint is memory. Give the box more RAM and the “disk problem” disappears.

# si and so are swap-in and swap-out in KB/s.
# Sustained non-zero values are your answer.
vmstat 1

# Are we thrashing writeback? Dirty pages waiting to be written.
grep -E 'Dirty|Writeback' /proc/meminfo

# The direct question
cat /proc/pressure/memory

If memory pressure and I/O pressure rise together, fix memory first. Everything else is downstream.

There’s a related trap on the write side. The kernel buffers writes in page cache and flushes them in the background, governed by vm.dirty_background_ratio and vm.dirty_ratio. Cross the background threshold and flusher threads start writing. Cross the hard vm.dirty_ratio threshold and writing processes are forced to do writeback themselves, synchronously. That transition is brutal, and on a box with plenty of RAM and a modest disk the defaults let you accumulate gigabytes of dirty pages and then pay for all of them at once. Lowering the ratios makes writeback more frequent and less spiky, trading away a little write batching. Measure before you commit to it.

Failure family three: small synchronous writes

Throughput can be trivial and the system still crawls. A workload doing thousands of tiny fsync calls a second is bound by round-trip latency, not bandwidth, and no amount of extra megabytes per second will help.

Databases are the usual source, because durability has a cost and that cost is a flush. MySQL and MariaDB expose it through innodb_flush_log_at_trx_commit; PostgreSQL through synchronous_commit. Loosening either buys real throughput and trades away durability on power loss or crash. That can be a perfectly reasonable trade for a cache or an analytics replica, and a terrible one for anything holding orders or payments. Decide it explicitly, write down what you decided, and don’t let it be a thing somebody changed once at 2am.

The other common source is logging. Application logs, access logs, container JSON logs and a database sharing one volume will starve each other. Splitting hot write paths onto separate devices is unglamorous and works.

Failure family four: throttled, not saturated

A throttled device and a saturated device look almost identical from inside the guest: high await, growing queue, miserable latency. The difference is that no amount of tuning inside the box will fix a throttle. Three flavours are worth knowing.

• Cloud volume limits. Managed block storage gives you a baseline of IOPS and throughput, sometimes with a burst allowance on top. Burn the burst and you drop to baseline, and everything that was fine in testing falls over in production. The tell is a latency cliff that arrives at a consistent point into a workload rather than tracking load smoothly. On AWS, watch the volume-level CloudWatch metrics such as BurstBalance on burstable volume types and VolumeQueueLength; other providers publish equivalents. The fix is a volume class change, not a sysctl.
• cgroup limits you set yourself. On cgroup v2, io.max caps a cgroup at a given bytes-per-second or IOPS figure, and io.weight shares bandwidth proportionally under contention. A limit that made sense when it was written can become the bottleneck a year later, and nothing in the container’s own metrics announces it.
• Neighbours on shared hardware. On oversubscribed VPS plans your storage is shared, and someone else’s backup job is your latency spike. This is where cheap and expensive hosting genuinely differ. Budget providers like Contabo and InterServer are fine value for a lot of workloads, but if storage latency is load-bearing for you, look for plans with dedicated NVMe rather than shared pools, and test before you migrate anything that matters.

# Per-device I/O accounting for a cgroup, including any throttling
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.stat

# Is a cap set? An empty file means no limit on that cgroup.
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.max

One more signal while you’re here. In top, the st column is steal time, meaning the hypervisor gave your vCPU’s slice to someone else. Steal and iowait rising together on a shared host is a strong hint that the problem is the neighbourhood, not your configuration.

Failure family five: the latency isn’t in the block layer

Here’s the failure mode that wastes the most time, because every storage metric you check will look healthy.

iostat measures the block layer. It cannot see anything above it. So if your latency is in NFS, in a network filesystem, in an overlay or FUSE mount, in filesystem lock contention, or in a journal that’s serialising writes, the device will report sub-millisecond service times while your application waits tens of milliseconds per operation.

The diagnosis is the gap. Measure latency at the application’s layer and at the block layer, and if the two numbers disagree wildly, the problem lives in between.

# Block layer latency as a histogram. Look for a bimodal shape
# or a long tail rather than the average.
sudo biolatency-bpfcc -m 10 1      # Debian/Ubuntu naming
sudo /usr/share/bcc/tools/biolatency -m 10 1   # RHEL family

# Filesystem operations slower than 10ms, with the process and file
sudo ext4slower-bpfcc 10
sudo xfsslower-bpfcc 10

# Everything at the VFS layer, higher overhead, use briefly
sudo fileslower-bpfcc 10

If ext4slower is full of hundred-millisecond operations and biolatency shows everything completing in microseconds, you have your answer and it is not the disk. For NFS specifically, nfsiostat splits out RTT from execution time, which separates network cost from server cost. Install the tooling with bpfcc-tools on Debian and Ubuntu, or bcc-tools on RHEL, AlmaLinux and Rocky.

A working order for troubleshooting high I/O wait

On an unfamiliar box, this is the sequence I work through. Each step either finds the problem or eliminates a family.

1. Read /proc/pressure/io. If full is near zero, the stall is not hurting the whole system and you should widen the search before you tune storage.
2. Check /proc/pressure/memory at the same time. If it’s rising too, fix memory first.
3. Run iostat -xy 1 for thirty seconds. Note r_await, w_await and aqu-sz per device. Ignore %util.
4. List processes stuck in uninterruptible sleep: ps -eo state,pid,comm | awk '$1 ~ /^D/'. That’s your blocked set, and it names the workload.
5. Attribute the traffic with pidstat -d 1. Confirm the process you suspect is the process actually doing the I/O.
6. If await is high, decide saturated versus throttled. Check cgroup io.max, check provider volume metrics, check steal time.
7. If await is low but the application is slow, jump to the layer gap. Run biolatency and ext4slower together and compare.
8. Only now consider tuning: dirty ratios, I/O scheduler, flush settings, splitting volumes.

Common mistakes

• Reading iostat’s first report. It’s a since-boot average. Use -y, or discard the first block.
• Trusting %util on NVMe or cloud volumes. Parallel devices break the assumption it was built on.
• Alerting on a fixed %iowait threshold. It fires on healthy backups and stays silent during real stalls on busy hosts.
• Changing the I/O scheduler first. Going from mq-deadline to none on NVMe is sometimes right, but it’s a tuning step, not a diagnosis, and it fixes nothing if you’re throttled.
• Adding swap to fix swapping. More swap on a memory-starved box means more disk traffic, not less.
• Stopping at the block layer. If every storage metric looks clean and the app is still slow, you’re measuring at the wrong layer, not measuring a healthy system.
• Testing with the wrong benchmark. A sequential dd tells you nothing about a workload doing random 4K synchronous writes. Use fio and match the real access pattern.

Best practices worth the effort

• Record PSI continuously. Storage incidents are short and bursty. Without history you’re guessing about something that already ended.
• Capture a baseline while things are healthy. Save an iostat -xy sample and a PSI reading from a normal day. “Is 8ms bad?” is unanswerable without one.
• Keep sar running. The sysstat package can archive metrics on a schedule, which means sar -d can show you what happened at 3am. Enable it before you need it.
• Separate hot write paths. Database data, database logs and application logs on separate volumes stops one from starving the others.
• Cap container logs. An uncapped JSON log driver quietly filling the root volume is a recurring cause of host-wide stalls.
• Alert on latency and pressure, not utilisation. Users feel w_await and PSI full. Nobody has ever felt a percentage of device busy time.

Frequently asked questions

What is a normal or acceptable I/O wait percentage?

There isn’t one, and any article that gives you a single number is guessing. Iowait scales with how idle your CPUs are, so the same workload produces very different figures on a busy server and a quiet one. Judge against your own baseline, and prefer PSI, which measures lost time directly rather than as a share of idle.

Can high I/O wait be caused by something other than the disk?

Regularly. Memory pressure driving swap and page cache eviction is the most common non-disk cause. Network filesystems, hypervisor contention on shared hosts, and cgroup or cloud volume throttling all produce the same symptom. The block device being genuinely slow is one possibility among several.

Why does iostat show 100% utilisation when the disk isn’t busy?

Because %util measures the fraction of time at least one request was in flight, which made sense for devices that handled one request at a time. NVMe drives and cloud volumes process many requests in parallel, so they can be continuously occupied while nowhere near capacity. Read aqu-sz and the await columns instead.

How do I find which process is causing I/O wait?

Start with pidstat -d 1, which needs no special kernel settings. List processes in uninterruptible sleep with ps -eo state,pid,comm and filter for state D to see who is actually blocked. iotop -o gives a live view but needs kernel.task_delayacct=1 on recent kernels for its percentage columns to populate.

Does adding more RAM reduce I/O wait?

Often, yes, and it’s underrated. More RAM means a larger page cache, fewer reads reaching the device, and less swapping. If vmstat shows sustained swap activity or memory pressure tracks I/O pressure, RAM is likely the cheapest fix available.

Should I change the I/O scheduler to fix high I/O wait?

Only after you know the device is the bottleneck. Check the current setting with cat /sys/block/sda/queue/scheduler. On NVMe, none is a common and sensible default because the hardware already does the queuing. On spinning disks, mq-deadline or bfq give better latency behaviour. It’s a tuning knob, not a diagnostic, and it will not help if you’re being throttled.

Why is my application slow when iostat says storage is fine?

Because iostat only sees the block layer. Latency in NFS, FUSE, overlay filesystems, journal serialisation or filesystem locking is entirely invisible to it. Run ext4slower or xfsslower alongside biolatency: if the filesystem is slow and the device is fast, the gap between them is your problem.

The one thing worth remembering

High I/O wait is a symptom that points in a general direction, not a diagnosis. It’s a slice of idle time, averaged across cores, measured at one specific layer of the stack. It goes up when nothing is wrong and stays flat when something is very wrong.

Read PSI to find out whether anyone is actually suffering. Read r_await, w_await and aqu-sz to find out whether the device is the constraint. Compare filesystem latency against block latency to find out whether you’re even looking at the right layer. Do those three things and the answer usually falls out in a few minutes, instead of after a week of arguing with your hosting provider about a volume that was fine all along.

Need someone to look at your storage stalls?

I work with teams on exactly this kind of problem, usually the ones that have already survived a round of “the host says the disk is fine”. Things I can help with:

• Diagnosing a live or recurring I/O stall and identifying which failure family it belongs to
• Setting up PSI, sar and per-cgroup I/O metrics in Prometheus and Grafana, with alerts that fire on latency rather than utilisation
• Separating cloud volume throttling from genuine device saturation, and sizing storage correctly instead of guessing upward
• Tuning writeback, dirty ratios and I/O schedulers for a specific workload, with before-and-after measurements
• Tracing container and Kubernetes I/O contention through cgroup v2 limits, overlay filesystems and log drivers
• Benchmarking with fio against your real access pattern so capacity decisions rest on something other than a dd run

If you’ve got an iostat capture, a PSI reading or just a graph that doesn’t make sense, send it over and I’ll tell you what I see in it.

The post Troubleshooting High I/O Wait Without Blaming the Disk appeared first on John Nessime.

Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker’s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.

This post covers Docker log rotation properly: what the defaults actually do, why the daemon.json change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.

The default is unbounded, and that is documented behaviour

Docker’s default logging driver is json-file. It captures stdout and stderr from the container’s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.

The important part is the option defaults. For json-file, max-size defaults to unlimited, max-file defaults to 1, and compress defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.

Docker’s own documentation is upfront about why. Keeping json-file unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.

Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:

# What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"

The gap between the first command and the second is where most of the pain lives, and we will come back to it.

To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container’s log file plus a small amount of metadata, so the sizes are close enough to be useful:

sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10

Container IDs are not memorable, so map the winner back to a name with docker ps --no-trunc or by grepping the ID against docker ps -aq. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.

Setting up Docker log rotation on the daemon

The daemon-wide fix goes in /etc/docker/daemon.json. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

What each key buys you:

• max-size caps a single file. Accepts an integer plus k, m or g. This is the setting that stops the runaway.
• max-file caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is only effective if max-size is also set, which is the single most common misconfiguration I see.
• compress gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.

Two things trip people up here. First, every value in log-opts must be a JSON string, including the numeric ones. Writing "max-file": 3 without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed daemon.json on a production host is a much worse afternoon than a full disk:

# Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  && echo "daemon.json parses OK"

sudo systemctl restart docker

Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is max-size multiplied by max-file. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.

Why your daemon.json change did nothing

This is the failure mode that costs the most time, because it looks exactly like the fix not working.

Logging configuration is baked into a container’s host config when the container is created. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because docker restart stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.

So the container that filled your disk keeps filling your disk, its log file still has no cap, and docker info cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.

To actually apply it, recreate the container and then verify rather than assuming:

# Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web

You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.

Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.

Pinning limits per service in Compose

Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides daemon.json entirely. A YAML anchor keeps it from turning into copy-paste sprawl:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

Be aware that the logging block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with its defaults, which for json-file means straight back to unbounded. Half-specifying is worse than not specifying at all.

You deleted the log file and df did not move

Classic 3am mistake, and it is worth understanding rather than memorising.

When you rm the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from ls, the daemon keeps writing into a file you can no longer see, and df reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.

truncate -s 0 is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:

sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"

There is a cost. Truncating out from under the daemon can leave an in-flight docker logs -f stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.

The same reasoning applies to logrotate rules pointed at /var/lib/docker/containers/*/*.log. Rotating with copytruncate works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker’s own docs warn against external tools touching those files. If the driver can rotate for you, let it.

Choosing a driver, and what each one costs

json-file

The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. docker logs works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.

local

Docker’s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it rotates and compresses by default, retaining five files of 20 MB each per container out of the box. docker logs works exactly as normal. The trade-off is that the format is Docker’s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is docker logs plus a shipper that talks to the daemon, this is the safest default you can pick.

journald

Hands container output to systemd’s journal, which means you get journalctl filtering, structured fields, and the journal’s own size management via SystemMaxUse in journald.conf. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.

The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at RateLimitIntervalSec and RateLimitBurst before you need them, not after.

syslog, fluentd and the remote drivers

These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.

What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.

none

Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because docker logs returns nothing and you will waste twenty minutes concluding the container is broken.

The blocking behaviour worth knowing about

By default the logging path is blocking. If the driver cannot keep up, the container’s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.

docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1

Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.

Dual logging, and why docker logs still works

If you switch to a remote driver and find docker logs still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the local driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.

Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.

Reclaiming space when the disk is already full

Order matters here. Free space first, fix the cause second, verify third.

1. Confirm logs are actually the problem. Compare du -sh /var/lib/docker/containers against docker system df, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.
2. Identify the largest log files with the du and sort command from earlier.
3. Copy anything you need out first, if the logs are part of an active investigation.
4. Truncate the offenders with truncate -s 0. Never rm.
5. Set the limits in daemon.json, validate the JSON, restart the daemon.
6. Recreate the containers so the limits take effect, then confirm with docker inspect on each one.
7. Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.

Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.

Troubleshooting

Rotation is configured but files still grow past the cap. The container predates the change. Check docker inspect --format '{{.HostConfig.LogConfig}}' and recreate it.

The daemon will not start after editing daemon.json. Almost always invalid JSON, or unquoted values in log-opts. Read the actual error with journalctl -u docker --no-pager -n 50 rather than guessing.

max-file is set but only one file ever appears. max-file does nothing without max-size. Set both.

docker logs -f hangs after showing existing output. Something truncated the file underneath the daemon, usually an external logrotate rule. Reattach to recover, then remove the rule.

docker logs returns nothing at all. Check the driver. none discards everything, and some remote drivers only serve reads through the dual-logging cache.

Disk usage did not drop after deleting log files. The daemon still holds the descriptors. Restart the daemon to release the inodes, and use truncate next time.

Logs vanish sooner than the retention you configured. On journald, check journal rate limiting and SystemMaxUse. On a remote driver in non-blocking mode, check whether the buffer is overflowing.

Common mistakes

• Assuming a daemon restart applies the new limits to running containers. It does not.
• Setting max-file without max-size and believing rotation is on.
• Writing numeric values unquoted in log-opts.
• Overwriting an existing daemon.json instead of merging, and silently dropping storage or network settings.
• Using rm on a live container’s log file and concluding the disk report is broken.
• Specifying a logging.driver in Compose without its options, which resets that service back to the driver’s own unbounded defaults.
• Pointing host logrotate at the Docker containers directory when the driver could have handled it.
• Sizing rotation per container without multiplying by container count against the actual disk.

Best practices

• Set logging limits in daemon.json as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.
• Prefer the local driver on standalone hosts unless something specifically needs to read the JSON files.
• Turn on compress. Log text compresses well enough that you get more retention for the same disk.
• Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.
• Do the multiplication. Worst-case host usage is max-size × max-file × container count, and it should be a fraction of your disk, not most of it.
• Alert on Docker data root usage at a threshold that leaves you time to act.
• Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.
• Verify with docker inspect after any change. docker info tells you the daemon’s intent, not what your containers carry.

Frequently asked questions

Does Docker rotate container logs automatically?

Not with the default driver. json-file ships with max-size unlimited and max-file set to 1, so a single file grows without bound. The local driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.

Where are Docker container logs stored on the host?

Under the Docker data root, one directory per container, which on a standard Linux install means /var/lib/docker/containers/. Rather than assembling the path by hand, ask the daemon: docker inspect --format '{{.LogPath}}' <container>. That works regardless of a custom data-root.

How do I clear Docker logs without stopping the container?

Truncate the file to zero bytes with truncate -s 0 on the path from {{.LogPath}}. Do not use rm: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.

Why did my daemon.json rotation settings not apply?

Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and docker restart reuses the same container object. Recreate the container, then confirm with docker inspect --format '{{.HostConfig.LogConfig}}'.

Should I use the local driver or json-file?

Use local if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use json-file if an agent or shipper on the host tails those files, because it cannot parse the local format. Either way docker logs behaves the same.

Can I use logrotate for Docker container logs instead?

You can, and plenty of hosts do, but the driver options are the better tool. A copytruncate rule pointed at the containers directory reaches into files the daemon treats as private, and can leave docker logs -f streams stalled after each rotation. Use logrotate for logs your applications write to mounted volumes, and let the driver handle stdout.

How much disk should I budget for container logs?

Worst case is max-size × max-file per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.

The one thing worth remembering

Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.

So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with docker inspect rather than trusting docker info. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.

Need a hand with container logging on your hosts?

Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:

• Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits
• Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number
• Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back
• Choosing between json-file, local, journald and shipping off-box, including the blocking behaviour trade-offs under load
• Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends
• Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot

If you want a second opinion, send me your daemon.json, the output of docker ps and a du -sh of your containers directory, and I will tell you what I would change.

The post Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation appeared first on John Nessime.

Twenty seconds later the underlying disk no longer exists in AWS. Not detached. Deleted. The reclaim policy on the StorageClass was Delete, which is the default on essentially every managed cluster, and deleting the claim cascaded straight through to the volume.

That is the sharp edge of Kubernetes persistent volumes, and it is a naming problem as much as a technical one. Three objects sit between your pod and a real disk, the relationships between them are not obvious from the names, and the defaults are tuned for ephemeral test clusters rather than for anything holding data you care about.

This is short and covers the parts that bite: reclaim policies, what access modes actually mean, why volumes pin pods to a zone, and the difference between a StatefulSet and a Deployment that matters in practice.

Three objects, one disk

The mental model that prevents most of this:

• StorageClass is the recipe. It says which driver provisions volumes and what the defaults are.
• PersistentVolume is the actual piece of storage in the cluster. Usually created for you rather than by you.
• PersistentVolumeClaim is the request a pod makes, and the thing that binds one-to-one with a PV.

The PVC is the object your team touches, and it is the one whose name most understates its power. Deleting a PVC can delete a disk. That behaviour comes from the StorageClass, so check what yours says before you need to know:

# Read the RECLAIMPOLICY column. On managed clusters it usually
# says Delete, and nothing anywhere warns you about that.
kubectl get storageclass

For anything holding real data, define your own class with Retain and use it deliberately:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-retain
provisioner: ebs.csi.aws.com
reclaimPolicy: Retain              # PV outlives the PVC
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3

The trade-off is real and worth stating: Retain means orphaned volumes accumulate and quietly cost money, and a released PV will not automatically rebind to a new claim. You are choosing a cleanup chore over a data loss risk. That is the right trade for a database and the wrong one for a scratch cache.

Existing volumes keep whatever policy they were created with, so changing the class does nothing retroactively. Patch them individually:

kubectl patch pv "$PV_NAME" 
  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

ReadWriteOnce does not mean one pod

It means one node. Multiple pods scheduled onto the same node can mount the same ReadWriteOnce volume and write to it simultaneously. The Kubernetes documentation says this explicitly, and almost everyone reads the name and assumes otherwise.

For a database, two writers on one filesystem is corruption. If you need genuine exclusivity, there is an access mode for it, stable since Kubernetes 1.29 and available on CSI volumes:

spec:
  accessModes:
    - ReadWriteOncePod        # enforced by the scheduler

The same access mode causes a second, more visible problem. Put a ReadWriteOnce volume on a Deployment with the default rolling update strategy and you get a deadlock: the new pod cannot start because the volume is still attached to the old pod’s node, and the old pod will not terminate until the new one is ready. The rollout sits there until it times out.

spec:
  strategy:
    type: Recreate            # old pod goes first, then the new one

That buys you downtime on every deploy, which is the honest cost of a single-writer volume behind a Deployment. If you cannot accept the downtime, you need either shared storage that supports ReadWriteMany or an application that handles its own replication, which is usually a StatefulSet.

Volumes pin pods to a zone

Block storage from a cloud provider lives in one availability zone. A pod using it can only ever be scheduled onto a node in that zone. Nothing in your Deployment spec says so, and the symptom is a pod stuck in Pending after a node replacement or a scale-up that happened to land elsewhere.

kubectl describe pod tells you, in among the scheduler events, that no node matched the volume’s node affinity. It is easy to miss if you are scanning for something more dramatic.

The preventive setting is volumeBindingMode: WaitForFirstConsumer, which is in the StorageClass above. It delays provisioning until a pod is actually scheduled, so the volume gets created in a zone where the pod can run rather than the other way round. Set it on every class you create. The immediate-binding alternative provisions a volume somewhere convenient and then constrains your scheduler forever.

StatefulSet or Deployment

The practical difference is not ordering guarantees or DNS names, useful as those are. It is that a StatefulSet gives each replica its own volume, created from a template, with a stable name that survives rescheduling. A Deployment with a PVC gives every replica the same volume, which is only correct when you have one replica.

So: one instance with a disk can be a Deployment with Recreate. Anything that scales and keeps per-replica state should be a StatefulSet.

StatefulSet PVCs historically survived everything, including deleting the StatefulSet, which is safe and leaves litter. Since Kubernetes 1.32 you can state the policy explicitly, and both fields still default to Retain:

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain       # deleting the StatefulSet keeps the data
    whenScaled: Delete        # scaling down cleans up the extra replicas

That pairing is the sensible default for most workloads. Be careful with whenScaled: Delete on anything where you scale down temporarily and expect the data back when you scale up again.

Two one-way doors

Volumes grow, they do not shrink. Expansion is stable and works by editing the claim’s requested size, provided the StorageClass allows it and the driver supports it. There is no supported path back. Over-provisioning a PVC is a permanent decision until you migrate the data to a smaller one by hand.

A volume snapshot is not a database backup. Snapshotting a running database’s disk gives you a crash-consistent image: the equivalent of pulling the power cable. Most engines recover from that, and “most” is not a word you want in a recovery plan. Use the database’s own backup mechanism for the database, and treat volume snapshots as infrastructure convenience rather than your restore path. Whichever you choose, restore one before you need to.

Troubleshooting

Pod stuck in Pending. Read the events at the bottom of kubectl describe pod. Volume node affinity conflict means a zone mismatch. No storage class means the cluster has no default and the claim did not name one.

PVC stuck in Pending. kubectl describe pvc names the provisioner error. With WaitForFirstConsumer this is normal until a pod is scheduled, so check whether anything is actually trying to consume it before debugging the driver.

Rollout hangs forever on a stateful app. ReadWriteOnce plus rolling update. Switch to Recreate.

PVC will not delete. A finalizer is holding it because something still mounts it. Find the consuming pod rather than removing the finalizer by hand, which is how you end up with a PV bound to a claim that no longer exists.

Volume is full but the PVC says it has room. You expanded the PVC and the filesystem did not follow. Most CSI drivers resize the filesystem on the next pod restart; if yours has not, restart the pod and check again.

Common mistakes

• Running production data on the default StorageClass without checking its reclaim policy.
• Assuming ReadWriteOnce prevents two pods from writing.
• A ReadWriteOnce volume on a Deployment with rolling updates.
• Scaling a Deployment with a PVC past one replica.
• Leaving volumeBindingMode: Immediate, then debugging Pending pods for an hour.
• Treating volume snapshots as database backups.
• Over-sizing a PVC on the assumption you can shrink it later.
• Removing a stuck finalizer instead of finding what is holding the volume.
• Deleting a namespace without checking what claims live in it.

Best practices

• Your own StorageClass with Retain for anything you would miss, and the default for everything else.
• WaitForFirstConsumer on every class you define.
• allowVolumeExpansion: true from the start, because adding it later does not help existing volumes.
• ReadWriteOncePod where exactly one writer is a correctness requirement.
• StatefulSets for per-replica state, single-replica Deployments with Recreate for everything else.
• An explicit persistentVolumeClaimRetentionPolicy rather than relying on the default.
• Application-level backups for databases, tested by restoring one.
• Alerts on volume capacity, because a full disk is the most predictable outage there is.

FAQ

Does deleting a PVC delete my data?

If the reclaim policy is Delete, yes, and that is the default nearly everywhere. Run kubectl get storageclass and read the RECLAIMPOLICY column before you assume otherwise.

Can two pods share a ReadWriteOnce volume?

Yes, if they are on the same node. ReadWriteOnce restricts access to one node, not one pod. Use ReadWriteOncePod when you need a single writer enforced.

Why is my pod Pending with a bound PVC?

Almost always zone affinity: the volume is in one availability zone and no schedulable node is there. The events in kubectl describe pod say so. WaitForFirstConsumer prevents it happening again.

Should I run databases in Kubernetes at all?

You can, with a mature operator that handles failover, backups and upgrades. Whether you should depends on whether your team wants to operate a database platform or just use a database. A managed service is often the better answer and there is no shame in it.

Can I shrink a PersistentVolumeClaim?

No. Expansion is supported, reduction is not. The only route down is a new smaller volume and a data migration.

The one thing to remember

Kubernetes storage defaults assume your cluster is disposable. Reclaim policies delete disks, access modes permit more sharing than their names suggest, and volumes constrain scheduling in ways nothing surfaces until a pod will not start.

Go and run kubectl get storageclass on your production cluster now. If the reclaim policy on the class holding your database says Delete, you have found something worth fixing before lunch.

Need a hand with stateful workloads?

Storage is where most Kubernetes migrations get uncomfortable, and it is usually the last thing anyone designs. Work I take on:

• Auditing a cluster’s storage classes, reclaim policies and access modes, and telling you what a stray kubectl delete would actually destroy.
• Moving stateful applications onto StatefulSets with sane retention, expansion and scheduling behaviour.
• Backup and restore design for data in Kubernetes, including an actual restore test rather than a documented intention.
• Diagnosing Pending pods, stuck rollouts and volumes that will not detach.
• CSI driver setup and storage class design on EKS and self-managed clusters.
• Capacity alerting so a full volume is a notification instead of an incident.

Send me the output of kubectl get storageclass and kubectl get pv, and I will tell you what stands out.

The post It’s Only a Claim: Kubernetes Persistent Volumes and Stateful Workloads appeared first on John Nessime.

Docker disk usage grows quietly because every part of it accumulates independently — images, stopped containers, volumes, build cache, and container logs, which are the one nobody expects and which no Docker command reports on. Cleaning it up is easy. Cleaning it up without destroying something takes about five more minutes.

This is the order I work through it: measure first, reclaim from safest to most destructive, then fix the thing that caused the growth so you are not back here next month.

Measure before you delete

Every cleanup starts here, and skipping it is how people delete the wrong thing:

docker system df

That gives you four rows — Images, Containers, Local Volumes, Build Cache — with a total size and a reclaimable figure for each. The reclaimable column is the one that matters. If build cache is eleven gigabytes and all of it is reclaimable, you have found your win and you never need to go near volumes.

For the itemised version:

docker system df -v

Now you can see individual images with their sizes, which containers are using what, and which volumes have no container attached. Read this before running anything destructive.

Two more views worth having open:

# What is actually running, and what is just sitting there stopped
docker ps -a --format 'table {{.Names}}t{{.Image}}t{{.Status}}t{{.Size}}'

# Images by age, oldest first
docker images --format 'table {{.Repository}}:{{.Tag}}t{{.Size}}t{{.CreatedSince}}'

Reclaim in order, safest first

These are ordered deliberately. Work down the list and stop as soon as you have enough space. Most of the time you never reach step four.

1. Stopped containers

# See what would go
docker ps -a --filter status=exited --filter status=created

docker container prune

This removes stopped containers and their writable layers. Anything running is untouched, and named volumes attached to those containers survive — only the container’s own layer goes.

The one caveat: if a stopped container held state in its writable layer rather than in a volume, that state is gone. That is a design problem rather than a cleanup problem, but it is worth a glance before you confirm.

2. Build cache

On any machine that builds images regularly, this is usually the biggest single win and it is completely safe. Build cache is derived data. Deleting it costs you a slower next build and nothing else.

# Everything older than a week
docker builder prune --filter "until=168h"

# Everything, no exceptions
docker builder prune -a

3. Dangling images

# Look first
docker images -f dangling=true

docker image prune

Without -a, this removes only dangling images — untagged layers left behind when a rebuild moved a tag to a new image. They are almost always genuine garbage.

4. Unused images — read this one carefully

docker image prune -a

Adding -a changes the meaning completely. It removes every image that does not have at least one container associated with it — running or stopped. So an image you pulled for a job that runs weekly, with no container currently existing, is gone. It will pull again, which is fine if you have bandwidth and the registry is reachable, and much less fine at 3am on a machine with a slow link.

Safer in most cases is an age filter, which keeps anything recent:

docker image prune -a --filter "until=336h"   # older than 14 days

5. Volumes — the one that loses data

Volumes are where databases live. Treat this step as a data operation, not a cleanup.

Before removing anything, look inside the candidates. This mounts each dangling volume read-only into a throwaway container so you can see what is actually in it:

docker volume ls -qf dangling=true | while read -r v; do
  echo "=== $v"
  docker run --rm -v "$v":/vol:ro alpine sh -c 'du -sh /vol; ls -la /vol' 2>/dev/null | head -12
done

If you see anything resembling a data directory, back it up before it goes:

docker run --rm 
  -v my-volume:/vol:ro 
  -v "$PWD":/backup 
  alpine tar czf /backup/my-volume.tar.gz -C /vol .

Only then:

docker volume prune

On recent Docker versions this prunes only anonymous volumes by default, with --all needed to include named ones — a genuinely good safety change. Older versions removed named volumes too. Check which behaviour you have before trusting it:

docker version --format '{{.Server.Version}}'
docker volume prune --help

The gigabytes Docker never tells you about

Here is the part that catches experienced people. Container logs do not appear in docker system df at all, and no prune command touches them. A chatty application with the default logging driver will happily write tens of gigabytes and nothing in Docker’s own tooling will mention it.

sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -10

To map a log file back to the container that owns it:

docker ps -aq | xargs docker inspect --format '{{.Name}} {{.LogPath}}'

Truncate, do not delete

This matters. Deleting a log file that a running container still has open removes the directory entry but not the data — the kernel keeps the blocks allocated until the process closes the descriptor, so df shows no improvement and you are left confused. Truncate instead:

sudo truncate -s 0 /var/lib/docker/containers/<container-id>/<container-id>-json.log

Then stop it happening again

Set a global default in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

sudo systemctl restart docker

One thing people get wrong here: this applies to newly created containers only. Anything already running keeps the logging config it was created with. You have to recreate those containers for the limit to take effect — restarting them is not enough.

While you are in that file, cap the build cache too:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "20GB"
    }
  }
}

Automating it without automating a disaster

A weekly cron is worth having, provided it only ever does the safe operations. The rule is simple: never put --volumes in anything unattended.

# /etc/cron.d/docker-prune
# Weekly, Sunday 03:00. Containers, networks, dangling images and build
# cache older than seven days. No volumes, ever.
0 3 * * 0 root /usr/bin/docker system prune -f --filter "until=168h" >> /var/log/docker-prune.log 2>&1

Note what docker system prune covers without flags: stopped containers, unused networks, dangling images and build cache. It does not touch volumes unless you explicitly pass --volumes, and it does not remove tagged images unless you pass -a. Those two flags are the whole difference between a routine job and an incident.

If you are tempted to add --volumes to a cron job so you do not have to think about it again, you have swapped a disk space problem for a data loss problem on a timer.

Troubleshooting

Pruned everything, df has not moved

Deleted files still held open by a running process. Find them:

sudo lsof +L1 | head -20

Restarting the process that holds the descriptor releases the blocks. If it is the Docker daemon itself, a systemctl restart docker does it, at the cost of bouncing every container without a restart policy.

“No space left on device” but df shows free space

You are out of inodes rather than bytes. Docker’s layered storage creates enormous numbers of small files:

df -i

The fix is the same — prune images and build cache, which is where the file count lives.

overlay2 is huge and I have hardly any images

sudo du -sh /var/lib/docker/*
sudo du -sh /var/lib/docker/overlay2 2>/dev/null

Usually build cache or container writable layers rather than images. Do not delete anything under overlay2 by hand — the directory names are referenced in Docker’s metadata, and removing them directly corrupts the daemon’s view of the world. Always go through docker commands.

Running low and need space right now

Fastest safe sequence, in order:

docker builder prune -a -f
docker container prune -f
docker image prune -f
sudo truncate -s 0 /var/lib/docker/containers/*/*-json.log

Four commands, no volumes touched, no tagged images removed. That resolves the large majority of Docker disk usage emergencies.

Common mistakes

• Reaching for docker system prune -a --volumes first because it is the top search result.
• Putting --volumes in a cron job.
• Deleting log files instead of truncating them, then wondering why df did not change.
• Setting log limits in daemon.json and assuming existing containers pick them up. They do not.
• Removing directories under /var/lib/docker/overlay2 by hand.
• Storing application data in a container’s writable layer rather than a volume, so cleanup destroys it.
• Never running docker system df, and so deleting the wrong category entirely.
• Assuming inodes cannot be the problem.

Best practices

• Set log rotation in daemon.json on every host, on day one.
• Cap build cache with the builder GC settings rather than relying on manual pruning.
• Use named volumes for anything you care about, so cleanup and data are clearly separated.
• Use multi-stage builds and a real .dockerignore. Smaller images mean less to clean up.
• Automate only the safe operations, with an age filter, and log what the job removed.
• Alert on disk usage at 80% so cleanup is a scheduled task rather than an incident.
• On busy build hosts, give /var/lib/docker its own partition so a runaway cache cannot take the OS down with it.

Frequently asked questions

Is docker system prune safe to run?

Without flags, yes, on most systems. It removes stopped containers, unused networks, dangling images and build cache. Adding -a also removes tagged images with no container attached. Adding --volumes is the one that can lose data.

Will pruning delete my database?

Only if the database lives in a volume and you pass --volumes, or it lives in a container’s writable layer and you prune that container. If it is in a named volume attached to a running container, ordinary pruning will not touch it.

Why does docker system df not match du?

Two reasons. It does not count container logs at all, and it accounts for shared image layers once rather than per image. du -sh /var/lib/docker is the honest total.

How do I stop logs growing without limit?

Set max-size and max-file under log-opts in /etc/docker/daemon.json, restart the daemon, then recreate existing containers so they pick up the new configuration.

Can I move Docker’s storage to another disk?

Yes, with data-root in daemon.json. Stop Docker, copy /var/lib/docker across preserving ownership and extended attributes, point data-root at the new location, then start it again. Copy rather than move until you have confirmed it works.

Does pruning affect running containers?

No. Prune operations skip anything in use. The risk is always to things that are stopped, dangling or unattached — which is exactly why checking what is stopped before you prune is worth the thirty seconds.

Wrapping up

Docker disk usage is four categories that grow independently, plus a fifth — logs — that hides from Docker’s own accounting. Run docker system df, work down from build cache to volumes, and stop as soon as you have the space. The nuclear option exists, but reaching for it first is how people discover which of their volumes mattered.

Once the immediate problem is solved, spend ten minutes on log rotation and builder GC. Those two settings turn this from a recurring emergency into something you never think about again.

Need a hand with Docker in production?

I work with teams on container infrastructure — image sizes that got out of hand, build pipelines that cache badly, storage and logging defaults that were never set, and monitoring that catches a full disk before the alert does.

If you would rather not spend the evening on it, you can hire me on Upwork.

The post Cleaning Up Docker Disk Usage Safely appeared first on John Nessime.