Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker’s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.

This post covers Docker log rotation properly: what the defaults actually do, why the daemon.json change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.

The default is unbounded, and that is documented behaviour

Docker’s default logging driver is json-file. It captures stdout and stderr from the container’s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.

The important part is the option defaults. For json-file, max-size defaults to unlimited, max-file defaults to 1, and compress defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.

Docker’s own documentation is upfront about why. Keeping json-file unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.

Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:

# What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"

The gap between the first command and the second is where most of the pain lives, and we will come back to it.

To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container’s log file plus a small amount of metadata, so the sizes are close enough to be useful:

sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10

Container IDs are not memorable, so map the winner back to a name with docker ps --no-trunc or by grepping the ID against docker ps -aq. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.

Setting up Docker log rotation on the daemon

The daemon-wide fix goes in /etc/docker/daemon.json. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

What each key buys you:

• max-size caps a single file. Accepts an integer plus k, m or g. This is the setting that stops the runaway.
• max-file caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is only effective if max-size is also set, which is the single most common misconfiguration I see.
• compress gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.

Two things trip people up here. First, every value in log-opts must be a JSON string, including the numeric ones. Writing "max-file": 3 without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed daemon.json on a production host is a much worse afternoon than a full disk:

# Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  && echo "daemon.json parses OK"

sudo systemctl restart docker

Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is max-size multiplied by max-file. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.

Why your daemon.json change did nothing

This is the failure mode that costs the most time, because it looks exactly like the fix not working.

Logging configuration is baked into a container’s host config when the container is created. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because docker restart stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.

So the container that filled your disk keeps filling your disk, its log file still has no cap, and docker info cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.

To actually apply it, recreate the container and then verify rather than assuming:

# Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web

You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.

Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.

Pinning limits per service in Compose

Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides daemon.json entirely. A YAML anchor keeps it from turning into copy-paste sprawl:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

Be aware that the logging block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with its defaults, which for json-file means straight back to unbounded. Half-specifying is worse than not specifying at all.

You deleted the log file and df did not move

Classic 3am mistake, and it is worth understanding rather than memorising.

When you rm the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from ls, the daemon keeps writing into a file you can no longer see, and df reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.

truncate -s 0 is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:

sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"

There is a cost. Truncating out from under the daemon can leave an in-flight docker logs -f stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.

The same reasoning applies to logrotate rules pointed at /var/lib/docker/containers/*/*.log. Rotating with copytruncate works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker’s own docs warn against external tools touching those files. If the driver can rotate for you, let it.

Choosing a driver, and what each one costs

json-file

The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. docker logs works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.

local

Docker’s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it rotates and compresses by default, retaining five files of 20 MB each per container out of the box. docker logs works exactly as normal. The trade-off is that the format is Docker’s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is docker logs plus a shipper that talks to the daemon, this is the safest default you can pick.

journald

Hands container output to systemd’s journal, which means you get journalctl filtering, structured fields, and the journal’s own size management via SystemMaxUse in journald.conf. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.

The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at RateLimitIntervalSec and RateLimitBurst before you need them, not after.

syslog, fluentd and the remote drivers

These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.

What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.

none

Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because docker logs returns nothing and you will waste twenty minutes concluding the container is broken.

The blocking behaviour worth knowing about

By default the logging path is blocking. If the driver cannot keep up, the container’s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.

docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1

Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.

Dual logging, and why docker logs still works

If you switch to a remote driver and find docker logs still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the local driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.

Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.

Reclaiming space when the disk is already full

Order matters here. Free space first, fix the cause second, verify third.

1. Confirm logs are actually the problem. Compare du -sh /var/lib/docker/containers against docker system df, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.
2. Identify the largest log files with the du and sort command from earlier.
3. Copy anything you need out first, if the logs are part of an active investigation.
4. Truncate the offenders with truncate -s 0. Never rm.
5. Set the limits in daemon.json, validate the JSON, restart the daemon.
6. Recreate the containers so the limits take effect, then confirm with docker inspect on each one.
7. Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.

Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.

Troubleshooting

Rotation is configured but files still grow past the cap. The container predates the change. Check docker inspect --format '{{.HostConfig.LogConfig}}' and recreate it.

The daemon will not start after editing daemon.json. Almost always invalid JSON, or unquoted values in log-opts. Read the actual error with journalctl -u docker --no-pager -n 50 rather than guessing.

max-file is set but only one file ever appears. max-file does nothing without max-size. Set both.

docker logs -f hangs after showing existing output. Something truncated the file underneath the daemon, usually an external logrotate rule. Reattach to recover, then remove the rule.

docker logs returns nothing at all. Check the driver. none discards everything, and some remote drivers only serve reads through the dual-logging cache.

Disk usage did not drop after deleting log files. The daemon still holds the descriptors. Restart the daemon to release the inodes, and use truncate next time.

Logs vanish sooner than the retention you configured. On journald, check journal rate limiting and SystemMaxUse. On a remote driver in non-blocking mode, check whether the buffer is overflowing.

Common mistakes

• Assuming a daemon restart applies the new limits to running containers. It does not.
• Setting max-file without max-size and believing rotation is on.
• Writing numeric values unquoted in log-opts.
• Overwriting an existing daemon.json instead of merging, and silently dropping storage or network settings.
• Using rm on a live container’s log file and concluding the disk report is broken.
• Specifying a logging.driver in Compose without its options, which resets that service back to the driver’s own unbounded defaults.
• Pointing host logrotate at the Docker containers directory when the driver could have handled it.
• Sizing rotation per container without multiplying by container count against the actual disk.

Best practices

• Set logging limits in daemon.json as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.
• Prefer the local driver on standalone hosts unless something specifically needs to read the JSON files.
• Turn on compress. Log text compresses well enough that you get more retention for the same disk.
• Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.
• Do the multiplication. Worst-case host usage is max-size × max-file × container count, and it should be a fraction of your disk, not most of it.
• Alert on Docker data root usage at a threshold that leaves you time to act.
• Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.
• Verify with docker inspect after any change. docker info tells you the daemon’s intent, not what your containers carry.

Frequently asked questions

Does Docker rotate container logs automatically?

Not with the default driver. json-file ships with max-size unlimited and max-file set to 1, so a single file grows without bound. The local driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.

Where are Docker container logs stored on the host?

Under the Docker data root, one directory per container, which on a standard Linux install means /var/lib/docker/containers/. Rather than assembling the path by hand, ask the daemon: docker inspect --format '{{.LogPath}}' <container>. That works regardless of a custom data-root.

How do I clear Docker logs without stopping the container?

Truncate the file to zero bytes with truncate -s 0 on the path from {{.LogPath}}. Do not use rm: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.

Why did my daemon.json rotation settings not apply?

Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and docker restart reuses the same container object. Recreate the container, then confirm with docker inspect --format '{{.HostConfig.LogConfig}}'.

Should I use the local driver or json-file?

Use local if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use json-file if an agent or shipper on the host tails those files, because it cannot parse the local format. Either way docker logs behaves the same.

Can I use logrotate for Docker container logs instead?

You can, and plenty of hosts do, but the driver options are the better tool. A copytruncate rule pointed at the containers directory reaches into files the daemon treats as private, and can leave docker logs -f streams stalled after each rotation. Use logrotate for logs your applications write to mounted volumes, and let the driver handle stdout.

How much disk should I budget for container logs?

Worst case is max-size × max-file per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.

The one thing worth remembering

Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.

So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with docker inspect rather than trusting docker info. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.

Need a hand with container logging on your hosts?

Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:

• Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits
• Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number
• Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back
• Choosing between json-file, local, journald and shipping off-box, including the blocking behaviour trade-offs under load
• Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends
• Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot

If you want a second opinion, send me your daemon.json, the output of docker ps and a du -sh of your containers directory, and I will tell you what I would change.

The post Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation appeared first on John Nessime.

The worse version of this problem is the mirror image. You reboot a server to clear something, come back to read what led up to it, and the journal starts at the reboot. Everything before is gone. Not rotated, not compressed, not archived somewhere clever. Gone, because the journal was living in RAM and nobody ever told it otherwise.

Both of those are the same misunderstanding wearing different clothes: not knowing where your logs actually live. This post walks through the journald vs rsyslog split on a modern Linux server, what each one really stores, the three failure modes that bite in production, and how to decide which of the two is the copy you keep.

What actually happens when a process logs a line

On any systemd-based distribution, systemd-journald is the first stop for almost everything. It is not one option among several. It sits underneath.

• Anything a unit writes to stdout or stderr, because systemd wires those to the journal by default.
• Kernel messages, read from the kernel ring buffer.
• Classic syslog(3) calls from libc, which land on the /dev/log socket that journald owns.
• Native journal API calls from anything linked against libsystemd, which carry structured key-value fields instead of a flat string.
• Audit records, if Audit=yes is in effect.

journald writes all of that into indexed binary journal files. Then, on most server distributions, rsyslog gets a second copy of the same events and writes them out as text into /var/log/messages, /var/log/secure, /var/log/maillog and friends.

That handoff happens one of two ways, and which one your box uses matters more than most people expect. More on that below. First, the part that costs you an outage.

Before changing anything, look at the effective journald configuration rather than the file you think is authoritative. Distributions ship drop-ins under /usr/lib/systemd/journald.conf.d/ that quietly override upstream defaults:

# Print the merged configuration, including every drop-in, in load order
systemd-analyze cat-config systemd/journald.conf

This is the single most useful command in this whole post. Half the arguments about “what the default is” evaporate once you run it, because upstream systemd and your distribution frequently disagree about ForwardToSyslog.

Failure one: the journal that was never on disk

journald’s Storage= setting defaults to auto. That word does more work than it looks like. Under auto, journald writes to /var/log/journal/ only if that directory already exists. If it does not, journald falls back to /run/log/journal/, which is tmpfs. Memory. Wiped on reboot.

Nothing warns you. journalctl works fine, colours are pretty, filters work. You just silently have no history.

The quickest test is boot history. If the journal is persistent you see multiple boots; if it is volatile you see exactly one:

journalctl --list-boots

To make it persistent, create the directory, let systemd-tmpfiles apply the correct ownership and ACLs, then restart the daemon and flush anything still held in RAM:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush
journalctl --list-boots

The systemd-tmpfiles step is the one people skip. The journal directory needs specific group ownership and ACLs so that unprivileged users can read their own entries and members of the systemd-journal group can read everything. Creating the directory with a bare mkdir and walking away usually works, but it is the kind of “usually” that produces a confusing permissions ticket six months later.

If you want the behaviour to be explicit rather than inferred from a directory’s existence, set it in /etc/systemd/journald.conf.d/00-storage.conf:

[Journal]
Storage=persistent

A drop-in file is better than editing the main config, because package upgrades will not fight you over it.

Failure two: paying for the same log line twice

Once the journal is persistent, you are storing everything twice on a default server build: once as compressed binary in the journal, once as text under /var/log/. Two stores, two completely separate retention policies, neither of which knows the other exists.

The journal is capped by SystemMaxUse= and SystemKeepFree=. Left unset, they default to 10 percent and 15 percent of the filesystem respectively, and journald honours whichever is stricter. Recent systemd also caps those calculated defaults so they do not grow without bound on a large disk, which is worth knowing if you have ever wondered why a huge volume did not produce a proportionally huge journal.

The text copy is capped by logrotate, configured somewhere completely different, usually /etc/logrotate.d/rsyslog, on a weekly or daily schedule with its own rotate count.

Measure both before you tune either:

# What the journal is actually consuming on disk
journalctl --disk-usage

# What everything under /var/log costs, sorted, biggest last
sudo du -sh /var/log/* | sort -h

On a small VPS this stops being academic quickly. A modest instance from a provider like InterServer, Hetzner or DigitalOcean often has /var sharing a single root volume of 20 to 40 GB. A journal allowed to take 10 percent of that, plus four weeks of rotated text logs, plus a database, plus a container image cache, is how you end up paged at two in the morning for something that is not a real incident.

If you decide the text files are the copy you keep, put the journal on a short leash:

[Journal]
# Hard ceiling on total journal size
SystemMaxUse=200M
# Never let the journal push free space below this
SystemKeepFree=1G
# Discard entries older than this regardless of size
MaxRetentionSec=2day

Those three do different jobs and you generally want at least two of them. SystemMaxUse bounds the journal itself. SystemKeepFree protects the rest of the filesystem from the journal. MaxRetentionSec bounds it in time, which is what auditors and data-retention policies actually care about.

To reclaim space immediately without waiting for the next rotation:

# Trim archived journal files until the total falls below 500M
sudo journalctl --vacuum-size=500M

# Or trim by age instead
sudo journalctl --vacuum-time=7d

One caveat that surprises people: vacuuming only removes archived journal files. The currently active file is never deleted. If almost all your usage is in one large active file, run journalctl --rotate first, then vacuum.

Failure three: two rate limiters, both silent

This is the one that is invisible until it matters, and it is the real reason to understand the journald vs rsyslog relationship rather than treating them as interchangeable.

There are two independent rate limiters in the default pipeline, and both drop messages quietly.

journald’s limiter

journald applies RateLimitIntervalSec= and RateLimitBurst= per service, defaulting to 30 seconds and 10000 messages. Exceed the burst inside the interval and the rest of that service’s messages in that window are discarded. Not queued. Discarded.

The one piece of good news is that journald tells you, in the journal itself. Grep for it:

journalctl --grep="Suppressed" -n 50 --no-pager

A chatty web server or a container using the journald log driver will hit this. Rather than disabling rate limiting globally, override it for the one unit that needs it, using a drop-in:

# /etc/systemd/system/nginx.service.d/logging.conf
[Service]
LogRateLimitIntervalSec=30s
LogRateLimitBurst=50000

Per-unit values override journald.conf for that unit only, which is exactly what you want. Turning the global limiter off means one runaway process can fill your disk and take the box down. Turning it up for the single service you know is noisy is a bounded decision.

rsyslog’s limiter

If rsyslog is reading via the imjournal module, it applies its own rate limit on top, defaulting to 20000 messages per 600 seconds. A message can therefore survive journald’s limiter, land in the journal, and still never reach /var/log/messages or your remote log host.

That is the nasty case, because your local investigation finds the events but your centralised alerting never fired. Two people look at two sources and reach opposite conclusions about whether something happened.

rsyslog exposes counters for this through its statistics module, including how many messages it read from the journal, how many it submitted onward, and how many it discarded to rate limiting. Enable impstat if you run anything log-volume-sensitive; the discard counter is the number that tells you whether your pipeline is lying to you.

How rsyslog gets its copy: imjournal or imuxsock

There are two mechanisms, they behave differently under load, and running both at once is a classic source of duplicate lines.

imuxsock: journald pushes a copy to a socket

With ForwardToSyslog=yes in journald, journald writes a classic syslog-formatted copy of each message to a dedicated socket, and rsyslog reads it with imuxsock.

• Where it wins: simple, no state file, no second rate limiter, and no possibility of rsyslog reading back its own output and looping.
• Where it does not: you get the flat syslog view only. Structured journal fields such as the originating unit are not carried across.

imjournal: rsyslog pulls from the journal

imjournal reads journal files directly and keeps a state file so it can resume where it left off after a restart.

• Where it wins: structured fields survive the trip. If you want to route or template on the systemd unit name rather than parsing it out of a message string, this is the only option of the two.
• Where it does not: extra rate limiter to reason about, a state file that can drift, and a documented risk that a corrupted journal database causes rsyslog to re-read the same entries in a loop. rsyslog’s own documentation recommends using imuxsock instead unless you specifically need the structured data.

A representative imjournal load line, with the rate limit stated explicitly rather than left implicit:

module(load="imjournal"
       StateFile="imjournal.state"
       Ratelimit.Interval="600"
       Ratelimit.Burst="20000")

Writing the defaults out by hand is worth the two extra lines. The next person to debug missing messages will see the limiter exists instead of assuming there isn’t one.

To find out which mechanism your box is using right now:

grep -R "imjournal|imuxsock" /etc/rsyslog.conf /etc/rsyslog.d/ 2>/dev/null
grep -R "ForwardToSyslog" /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ /usr/lib/systemd/journald.conf.d/ 2>/dev/null

If the first command returns both modules and the second returns ForwardToSyslog=yes, you have a duplication problem. Pick one path and disable the other.

Only one of the two leaves the box

This is the argument that settles most journald vs rsyslog debates in practice. journald is a local store. In a stock server install it has no mechanism for shipping logs to another host. There are separate systemd components for journal upload and reception, and third-party projects that read the journal and forward it, but none of that is in the default path.

rsyslog forwards as a first-class feature, and it does the part that actually matters: it buffers when the destination is unreachable.

global(workDirectory="/var/spool/rsyslog")

action(type="omfwd"
       target="logs.example.net"
       port="514"
       protocol="tcp"
       action.resumeRetryCount="-1"
       queue.type="LinkedList"
       queue.filename="fwd_loghost"
       queue.maxDiskSpace="1g"
       queue.saveOnShutdown="on")

Line by line, because these are not decoration:

• workDirectory is where spool files get written. Without it, the disk queue has nowhere to go.
• protocol="tcp" rather than UDP. UDP syslog silently drops under congestion, which defeats the point of forwarding at all.
• queue.type="LinkedList" makes the action asynchronous so a slow destination does not block local processing.
• queue.filename is what actually enables disk assistance. In-memory queue first, spilling to disk when it fills. The name must be unique across every action in the config.
• queue.maxDiskSpace caps that spool. Set it, or a long outage at the far end fills the disk you were trying to protect.
• queue.saveOnShutdown="on" persists whatever is still queued across a service restart.

One honest caveat on action.resumeRetryCount="-1". Infinite retry is the right default for “never lose a log line”, but there are reported cases where a relay under sustained back-pressure stops accepting new inbound messages while retrying forever. If your host is a relay rather than a leaf, test that behaviour deliberately with the destination firewalled off before you trust it.

rsyslog is not the only way off the box. Grafana Alloy into Loki, Vector, and Fluent Bit all read the journal directly and speak modern backends, and hosted platforms such as Better Stack, Papertrail or Datadog will take a plain syslog stream if you would rather not run the storage side. The mechanism differs; the decision does not. Something has to own the write path off the machine, and journald on its own is not it.

journald vs rsyslog: how I would decide

Stop asking which is better. Ask which one is the copy you are willing to be judged on, then make the other one cheap.

1. Do logs need to leave this machine? If yes, rsyslog or a dedicated shipper owns the outbound path. That is settled before anything else.
2. Do you troubleshoot mostly by unit? If journalctl -u something -b is your muscle memory, journald is your primary read path and should get the retention budget.
3. Do existing tools read text files? Fail2ban, logwatch, older SIEM collectors and a lot of home-grown scripts parse /var/log/*. Turning rsyslog off breaks them quietly.
4. How much disk do you actually have? Under about 40 GB, keeping two full copies is an unforced error. Cap one hard.
5. Is tamper-evidence a requirement? journald supports Forward Secure Sealing, which detects after-the-fact modification of journal files. Shipping to an append-only remote store is the more commonly accepted answer, and the two are not mutually exclusive.

The configuration I reach for first on a general-purpose server: journald persistent with a hard cap for a few days of fast local querying, rsyslog kept for the syslog files that other tools expect plus reliable forwarding off-box, imuxsock as the handoff unless something genuinely needs structured fields, and logrotate retention trimmed down because the remote copy is the one that matters for anything older than a week.

Troubleshooting

• Journal only shows the current boot. Volatile storage. journalctl --list-boots confirms it, and the fix is the persistent-storage sequence above.
• Events in journalctl but not in /var/log/messages. Either rsyslog is not reading from the journal at all, or the imjournal rate limiter is discarding. Check which input module is loaded first.
• Every line appears twice in the text logs. Both imjournal and imuxsock are active while ForwardToSyslog=yes. Disable one path.
• Bursts of messages disappear during incidents. That is a rate limiter, and incidents are exactly when services get loud. Search for suppression notices and raise the limit for that specific unit.
• journalctl reports corruption or behaves oddly. Run journalctl --verify. If files are damaged, journalctl --rotate starts a fresh active file so new writes are clean, then vacuum the bad archives.
• “Journal has been rotated since unit was started.” Not an error. Rotation happened mid-session, so journalctl -u cannot map the full range. Re-run the query without the unit filter or widen the time range.
• Disk still full after vacuuming. Vacuum skips the active journal file. Rotate first, then vacuum again.

Common mistakes

• Assuming the journal is persistent because journalctl returns results. It always returns results. The question is how far back.
• Disabling rsyslog on a box where fail2ban, logwatch or a scraper still reads /var/log/secure. Nothing errors. Detection just stops.
• Setting RateLimitBurst=0 globally to “stop losing logs”, then having a crash-looping service fill the disk in an afternoon.
• Editing /etc/systemd/journald.conf directly and being surprised when a vendor drop-in overrides it. Use a drop-in of your own with a name that sorts later.
• Forwarding over UDP because it is the one-line version. It drops silently under exactly the load that produced the logs you wanted.
• Enabling a disk-assisted queue without queue.maxDiskSpace, turning a remote outage into a local disk-full outage.
• Tuning journald limits on a machine whose real problem is that one application logs every health check at info level. Fix the source first.

Best practices

• Make storage explicit. Storage=persistent or Storage=volatile, never left to auto on a server you care about.
• Always set SystemKeepFree alongside SystemMaxUse. The first bounds the journal, the second protects everything else on the volume from it.
• Decide consciously which store is authoritative, and write it in the runbook. Two stores with no stated owner means neither gets maintained.
• Use exactly one journal-to-rsyslog path. Both is duplication; neither is silent data loss.
• Override rate limits per unit, not globally.
• Monitor journalctl --disk-usage as a metric, not as something you check after the alert.
• Ship off-box with TCP and a bounded disk-assisted queue, and test it with the destination firewalled off before you rely on it.
• Keep the local copy short and the remote copy long. Local disk is the expensive place to store history.

Frequently asked questions

Can I just disable rsyslog and use journald only?

On a self-contained box with no remote logging and nothing parsing text files, yes, and it saves real disk. Before you do it, grep your configuration management and cron jobs for /var/log/ paths, and check whether fail2ban or any monitoring agent reads those files. That is where this bites people.

Can I disable journald and use rsyslog only?

Not meaningfully. journald is how systemd captures unit stdout and stderr, so it stays in the path regardless. What you can do is set Storage=volatile with a small RuntimeMaxUse so it holds a short in-memory window and rsyslog owns everything durable.

Where are journald logs stored?

/var/log/journal/ when persistent, /run/log/journal/ when volatile. They are indexed binary files, not text, so grep and tail do not work on them directly. Use journalctl, or journalctl -o json if you want to pipe structured output into something else.

Why does journalctl show entries that never reached /var/log/messages?

Three usual causes: rsyslog is not reading the journal at all, the imjournal rate limiter dropped them, or a severity filter such as MaxLevelSyslog or an rsyslog rule excluded that facility or priority. Check in that order.

Is the binary journal format a problem for log analysis?

Locally, no. Structured fields and fast filtering are genuinely better than parsing text with regex. It becomes a problem when you need to move logs somewhere else, because almost every collector expects text or JSON. That conversion step is precisely the role rsyslog or a modern shipper plays.

How much disk should the journal be allowed to use?

Work backwards from how far back you actually query locally. If you rarely look past yesterday, a few hundred megabytes with MaxRetentionSec set to two or three days is plenty. Anything older belongs in a remote store where it is cheaper and survives the machine.

Does journald compress logs automatically?

Yes. Compress= defaults to enabled and applies to entries above a size threshold, which is one reason the journal is often smaller than the equivalent text files despite carrying more metadata. It does not remove the need for a size cap.

The one thing worth remembering

The journald vs rsyslog question is not a contest. On a stock server both are running, both are storing the same events, and the defaults were chosen by your distribution rather than by you. The failure modes all come from that: a journal in RAM that vanishes on reboot, two uncoordinated retention policies eating the same disk, and two rate limiters dropping messages without anyone noticing.

Run systemd-analyze cat-config systemd/journald.conf and journalctl --list-boots on your servers. Two commands, thirty seconds, and you will know whether the logs you would reach for during an incident are actually there.

Need help sorting out your logging pipeline?

Logging is one of those areas where the defaults work well enough to hide the problem until an incident makes it expensive. If any of the above sounded familiar, these are the things I take on:

• Auditing journald and rsyslog on existing servers and telling you exactly where each log line lives, how long it survives, and what is being silently dropped.
• Fixing volatile journals, runaway /var growth and duplicate text-and-binary storage on small VPS instances.
• Setting up reliable off-box forwarding with disk-assisted queues, TCP or RELP transport, and TLS where it is required.
• Migrating from text-file scraping to a structured pipeline into Loki, Elasticsearch or a hosted log platform, without losing the alerts you already depend on.
• Tuning rate limits and retention per service so noisy applications stop hiding the messages that matter.
• Writing the runbook that says which store is authoritative, so the next engineer does not have to reverse-engineer it during an outage.

Send me your journald.conf, your rsyslog.conf, or the output of journalctl --disk-usage and I will tell you what I see before we talk about scope.

The post journald vs rsyslog: Where Your Linux Logs Actually Live appeared first on John Nessime.

Docker disk usage grows quietly because every part of it accumulates independently — images, stopped containers, volumes, build cache, and container logs, which are the one nobody expects and which no Docker command reports on. Cleaning it up is easy. Cleaning it up without destroying something takes about five more minutes.

This is the order I work through it: measure first, reclaim from safest to most destructive, then fix the thing that caused the growth so you are not back here next month.

Measure before you delete

Every cleanup starts here, and skipping it is how people delete the wrong thing:

docker system df

That gives you four rows — Images, Containers, Local Volumes, Build Cache — with a total size and a reclaimable figure for each. The reclaimable column is the one that matters. If build cache is eleven gigabytes and all of it is reclaimable, you have found your win and you never need to go near volumes.

For the itemised version:

docker system df -v

Now you can see individual images with their sizes, which containers are using what, and which volumes have no container attached. Read this before running anything destructive.

Two more views worth having open:

# What is actually running, and what is just sitting there stopped
docker ps -a --format 'table {{.Names}}t{{.Image}}t{{.Status}}t{{.Size}}'

# Images by age, oldest first
docker images --format 'table {{.Repository}}:{{.Tag}}t{{.Size}}t{{.CreatedSince}}'

Reclaim in order, safest first

These are ordered deliberately. Work down the list and stop as soon as you have enough space. Most of the time you never reach step four.

1. Stopped containers

# See what would go
docker ps -a --filter status=exited --filter status=created

docker container prune

This removes stopped containers and their writable layers. Anything running is untouched, and named volumes attached to those containers survive — only the container’s own layer goes.

The one caveat: if a stopped container held state in its writable layer rather than in a volume, that state is gone. That is a design problem rather than a cleanup problem, but it is worth a glance before you confirm.

2. Build cache

On any machine that builds images regularly, this is usually the biggest single win and it is completely safe. Build cache is derived data. Deleting it costs you a slower next build and nothing else.

# Everything older than a week
docker builder prune --filter "until=168h"

# Everything, no exceptions
docker builder prune -a

3. Dangling images

# Look first
docker images -f dangling=true

docker image prune

Without -a, this removes only dangling images — untagged layers left behind when a rebuild moved a tag to a new image. They are almost always genuine garbage.

4. Unused images — read this one carefully

docker image prune -a

Adding -a changes the meaning completely. It removes every image that does not have at least one container associated with it — running or stopped. So an image you pulled for a job that runs weekly, with no container currently existing, is gone. It will pull again, which is fine if you have bandwidth and the registry is reachable, and much less fine at 3am on a machine with a slow link.

Safer in most cases is an age filter, which keeps anything recent:

docker image prune -a --filter "until=336h"   # older than 14 days

5. Volumes — the one that loses data

Volumes are where databases live. Treat this step as a data operation, not a cleanup.

Before removing anything, look inside the candidates. This mounts each dangling volume read-only into a throwaway container so you can see what is actually in it:

docker volume ls -qf dangling=true | while read -r v; do
  echo "=== $v"
  docker run --rm -v "$v":/vol:ro alpine sh -c 'du -sh /vol; ls -la /vol' 2>/dev/null | head -12
done

If you see anything resembling a data directory, back it up before it goes:

docker run --rm 
  -v my-volume:/vol:ro 
  -v "$PWD":/backup 
  alpine tar czf /backup/my-volume.tar.gz -C /vol .

Only then:

docker volume prune

On recent Docker versions this prunes only anonymous volumes by default, with --all needed to include named ones — a genuinely good safety change. Older versions removed named volumes too. Check which behaviour you have before trusting it:

docker version --format '{{.Server.Version}}'
docker volume prune --help

The gigabytes Docker never tells you about

Here is the part that catches experienced people. Container logs do not appear in docker system df at all, and no prune command touches them. A chatty application with the default logging driver will happily write tens of gigabytes and nothing in Docker’s own tooling will mention it.

sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -10

To map a log file back to the container that owns it:

docker ps -aq | xargs docker inspect --format '{{.Name}} {{.LogPath}}'

Truncate, do not delete

This matters. Deleting a log file that a running container still has open removes the directory entry but not the data — the kernel keeps the blocks allocated until the process closes the descriptor, so df shows no improvement and you are left confused. Truncate instead:

sudo truncate -s 0 /var/lib/docker/containers/<container-id>/<container-id>-json.log

Then stop it happening again

Set a global default in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

sudo systemctl restart docker

One thing people get wrong here: this applies to newly created containers only. Anything already running keeps the logging config it was created with. You have to recreate those containers for the limit to take effect — restarting them is not enough.

While you are in that file, cap the build cache too:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "20GB"
    }
  }
}

Automating it without automating a disaster

A weekly cron is worth having, provided it only ever does the safe operations. The rule is simple: never put --volumes in anything unattended.

# /etc/cron.d/docker-prune
# Weekly, Sunday 03:00. Containers, networks, dangling images and build
# cache older than seven days. No volumes, ever.
0 3 * * 0 root /usr/bin/docker system prune -f --filter "until=168h" >> /var/log/docker-prune.log 2>&1

Note what docker system prune covers without flags: stopped containers, unused networks, dangling images and build cache. It does not touch volumes unless you explicitly pass --volumes, and it does not remove tagged images unless you pass -a. Those two flags are the whole difference between a routine job and an incident.

If you are tempted to add --volumes to a cron job so you do not have to think about it again, you have swapped a disk space problem for a data loss problem on a timer.

Troubleshooting

Pruned everything, df has not moved

Deleted files still held open by a running process. Find them:

sudo lsof +L1 | head -20

Restarting the process that holds the descriptor releases the blocks. If it is the Docker daemon itself, a systemctl restart docker does it, at the cost of bouncing every container without a restart policy.

“No space left on device” but df shows free space

You are out of inodes rather than bytes. Docker’s layered storage creates enormous numbers of small files:

df -i

The fix is the same — prune images and build cache, which is where the file count lives.

overlay2 is huge and I have hardly any images

sudo du -sh /var/lib/docker/*
sudo du -sh /var/lib/docker/overlay2 2>/dev/null

Usually build cache or container writable layers rather than images. Do not delete anything under overlay2 by hand — the directory names are referenced in Docker’s metadata, and removing them directly corrupts the daemon’s view of the world. Always go through docker commands.

Running low and need space right now

Fastest safe sequence, in order:

docker builder prune -a -f
docker container prune -f
docker image prune -f
sudo truncate -s 0 /var/lib/docker/containers/*/*-json.log

Four commands, no volumes touched, no tagged images removed. That resolves the large majority of Docker disk usage emergencies.

Common mistakes

• Reaching for docker system prune -a --volumes first because it is the top search result.
• Putting --volumes in a cron job.
• Deleting log files instead of truncating them, then wondering why df did not change.
• Setting log limits in daemon.json and assuming existing containers pick them up. They do not.
• Removing directories under /var/lib/docker/overlay2 by hand.
• Storing application data in a container’s writable layer rather than a volume, so cleanup destroys it.
• Never running docker system df, and so deleting the wrong category entirely.
• Assuming inodes cannot be the problem.

Best practices

• Set log rotation in daemon.json on every host, on day one.
• Cap build cache with the builder GC settings rather than relying on manual pruning.
• Use named volumes for anything you care about, so cleanup and data are clearly separated.
• Use multi-stage builds and a real .dockerignore. Smaller images mean less to clean up.
• Automate only the safe operations, with an age filter, and log what the job removed.
• Alert on disk usage at 80% so cleanup is a scheduled task rather than an incident.
• On busy build hosts, give /var/lib/docker its own partition so a runaway cache cannot take the OS down with it.

Frequently asked questions

Is docker system prune safe to run?

Without flags, yes, on most systems. It removes stopped containers, unused networks, dangling images and build cache. Adding -a also removes tagged images with no container attached. Adding --volumes is the one that can lose data.

Will pruning delete my database?

Only if the database lives in a volume and you pass --volumes, or it lives in a container’s writable layer and you prune that container. If it is in a named volume attached to a running container, ordinary pruning will not touch it.

Why does docker system df not match du?

Two reasons. It does not count container logs at all, and it accounts for shared image layers once rather than per image. du -sh /var/lib/docker is the honest total.

How do I stop logs growing without limit?

Set max-size and max-file under log-opts in /etc/docker/daemon.json, restart the daemon, then recreate existing containers so they pick up the new configuration.

Can I move Docker’s storage to another disk?

Yes, with data-root in daemon.json. Stop Docker, copy /var/lib/docker across preserving ownership and extended attributes, point data-root at the new location, then start it again. Copy rather than move until you have confirmed it works.

Does pruning affect running containers?

No. Prune operations skip anything in use. The risk is always to things that are stopped, dangling or unattached — which is exactly why checking what is stopped before you prune is worth the thirty seconds.

Wrapping up

Docker disk usage is four categories that grow independently, plus a fifth — logs — that hides from Docker’s own accounting. Run docker system df, work down from build cache to volumes, and stop as soon as you have the space. The nuclear option exists, but reaching for it first is how people discover which of their volumes mattered.

Once the immediate problem is solved, spend ten minutes on log rotation and builder GC. Those two settings turn this from a recurring emergency into something you never think about again.

Need a hand with Docker in production?

I work with teams on container infrastructure — image sizes that got out of hand, build pipelines that cache badly, storage and logging defaults that were never set, and monitoring that catches a full disk before the alert does.

If you would rather not spend the evening on it, you can hire me on Upwork.

The post Cleaning Up Docker Disk Usage Safely appeared first on John Nessime.