Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker’s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.

This post covers Docker log rotation properly: what the defaults actually do, why the daemon.json change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.

The default is unbounded, and that is documented behaviour

Docker’s default logging driver is json-file. It captures stdout and stderr from the container’s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.

The important part is the option defaults. For json-file, max-size defaults to unlimited, max-file defaults to 1, and compress defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.

Docker’s own documentation is upfront about why. Keeping json-file unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.

Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:

# What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"

The gap between the first command and the second is where most of the pain lives, and we will come back to it.

To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container’s log file plus a small amount of metadata, so the sizes are close enough to be useful:

sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10

Container IDs are not memorable, so map the winner back to a name with docker ps --no-trunc or by grepping the ID against docker ps -aq. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.

Setting up Docker log rotation on the daemon

The daemon-wide fix goes in /etc/docker/daemon.json. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

What each key buys you:

• max-size caps a single file. Accepts an integer plus k, m or g. This is the setting that stops the runaway.
• max-file caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is only effective if max-size is also set, which is the single most common misconfiguration I see.
• compress gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.

Two things trip people up here. First, every value in log-opts must be a JSON string, including the numeric ones. Writing "max-file": 3 without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed daemon.json on a production host is a much worse afternoon than a full disk:

# Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  && echo "daemon.json parses OK"

sudo systemctl restart docker

Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is max-size multiplied by max-file. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.

Why your daemon.json change did nothing

This is the failure mode that costs the most time, because it looks exactly like the fix not working.

Logging configuration is baked into a container’s host config when the container is created. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because docker restart stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.

So the container that filled your disk keeps filling your disk, its log file still has no cap, and docker info cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.

To actually apply it, recreate the container and then verify rather than assuming:

# Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web

You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.

Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.

Pinning limits per service in Compose

Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides daemon.json entirely. A YAML anchor keeps it from turning into copy-paste sprawl:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

Be aware that the logging block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with its defaults, which for json-file means straight back to unbounded. Half-specifying is worse than not specifying at all.

You deleted the log file and df did not move

Classic 3am mistake, and it is worth understanding rather than memorising.

When you rm the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from ls, the daemon keeps writing into a file you can no longer see, and df reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.

truncate -s 0 is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:

sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"

There is a cost. Truncating out from under the daemon can leave an in-flight docker logs -f stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.

The same reasoning applies to logrotate rules pointed at /var/lib/docker/containers/*/*.log. Rotating with copytruncate works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker’s own docs warn against external tools touching those files. If the driver can rotate for you, let it.

Choosing a driver, and what each one costs

json-file

The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. docker logs works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.

local

Docker’s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it rotates and compresses by default, retaining five files of 20 MB each per container out of the box. docker logs works exactly as normal. The trade-off is that the format is Docker’s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is docker logs plus a shipper that talks to the daemon, this is the safest default you can pick.

journald

Hands container output to systemd’s journal, which means you get journalctl filtering, structured fields, and the journal’s own size management via SystemMaxUse in journald.conf. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.

The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at RateLimitIntervalSec and RateLimitBurst before you need them, not after.

syslog, fluentd and the remote drivers

These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.

What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.

none

Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because docker logs returns nothing and you will waste twenty minutes concluding the container is broken.

The blocking behaviour worth knowing about

By default the logging path is blocking. If the driver cannot keep up, the container’s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.

docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1

Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.

Dual logging, and why docker logs still works

If you switch to a remote driver and find docker logs still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the local driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.

Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.

Reclaiming space when the disk is already full

Order matters here. Free space first, fix the cause second, verify third.

1. Confirm logs are actually the problem. Compare du -sh /var/lib/docker/containers against docker system df, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.
2. Identify the largest log files with the du and sort command from earlier.
3. Copy anything you need out first, if the logs are part of an active investigation.
4. Truncate the offenders with truncate -s 0. Never rm.
5. Set the limits in daemon.json, validate the JSON, restart the daemon.
6. Recreate the containers so the limits take effect, then confirm with docker inspect on each one.
7. Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.

Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.

Troubleshooting

Rotation is configured but files still grow past the cap. The container predates the change. Check docker inspect --format '{{.HostConfig.LogConfig}}' and recreate it.

The daemon will not start after editing daemon.json. Almost always invalid JSON, or unquoted values in log-opts. Read the actual error with journalctl -u docker --no-pager -n 50 rather than guessing.

max-file is set but only one file ever appears. max-file does nothing without max-size. Set both.

docker logs -f hangs after showing existing output. Something truncated the file underneath the daemon, usually an external logrotate rule. Reattach to recover, then remove the rule.

docker logs returns nothing at all. Check the driver. none discards everything, and some remote drivers only serve reads through the dual-logging cache.

Disk usage did not drop after deleting log files. The daemon still holds the descriptors. Restart the daemon to release the inodes, and use truncate next time.

Logs vanish sooner than the retention you configured. On journald, check journal rate limiting and SystemMaxUse. On a remote driver in non-blocking mode, check whether the buffer is overflowing.

Common mistakes

• Assuming a daemon restart applies the new limits to running containers. It does not.
• Setting max-file without max-size and believing rotation is on.
• Writing numeric values unquoted in log-opts.
• Overwriting an existing daemon.json instead of merging, and silently dropping storage or network settings.
• Using rm on a live container’s log file and concluding the disk report is broken.
• Specifying a logging.driver in Compose without its options, which resets that service back to the driver’s own unbounded defaults.
• Pointing host logrotate at the Docker containers directory when the driver could have handled it.
• Sizing rotation per container without multiplying by container count against the actual disk.

Best practices

• Set logging limits in daemon.json as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.
• Prefer the local driver on standalone hosts unless something specifically needs to read the JSON files.
• Turn on compress. Log text compresses well enough that you get more retention for the same disk.
• Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.
• Do the multiplication. Worst-case host usage is max-size × max-file × container count, and it should be a fraction of your disk, not most of it.
• Alert on Docker data root usage at a threshold that leaves you time to act.
• Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.
• Verify with docker inspect after any change. docker info tells you the daemon’s intent, not what your containers carry.

Frequently asked questions

Does Docker rotate container logs automatically?

Not with the default driver. json-file ships with max-size unlimited and max-file set to 1, so a single file grows without bound. The local driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.

Where are Docker container logs stored on the host?

Under the Docker data root, one directory per container, which on a standard Linux install means /var/lib/docker/containers/. Rather than assembling the path by hand, ask the daemon: docker inspect --format '{{.LogPath}}' <container>. That works regardless of a custom data-root.

How do I clear Docker logs without stopping the container?

Truncate the file to zero bytes with truncate -s 0 on the path from {{.LogPath}}. Do not use rm: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.

Why did my daemon.json rotation settings not apply?

Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and docker restart reuses the same container object. Recreate the container, then confirm with docker inspect --format '{{.HostConfig.LogConfig}}'.

Should I use the local driver or json-file?

Use local if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use json-file if an agent or shipper on the host tails those files, because it cannot parse the local format. Either way docker logs behaves the same.

Can I use logrotate for Docker container logs instead?

You can, and plenty of hosts do, but the driver options are the better tool. A copytruncate rule pointed at the containers directory reaches into files the daemon treats as private, and can leave docker logs -f streams stalled after each rotation. Use logrotate for logs your applications write to mounted volumes, and let the driver handle stdout.

How much disk should I budget for container logs?

Worst case is max-size × max-file per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.

The one thing worth remembering

Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.

So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with docker inspect rather than trusting docker info. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.

Need a hand with container logging on your hosts?

Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:

• Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits
• Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number
• Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back
• Choosing between json-file, local, journald and shipping off-box, including the blocking behaviour trade-offs under load
• Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends
• Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot

If you want a second opinion, send me your daemon.json, the output of docker ps and a du -sh of your containers directory, and I will tell you what I would change.

The post Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation appeared first on John Nessime.

The worse version of this problem is the mirror image. You reboot a server to clear something, come back to read what led up to it, and the journal starts at the reboot. Everything before is gone. Not rotated, not compressed, not archived somewhere clever. Gone, because the journal was living in RAM and nobody ever told it otherwise.

Both of those are the same misunderstanding wearing different clothes: not knowing where your logs actually live. This post walks through the journald vs rsyslog split on a modern Linux server, what each one really stores, the three failure modes that bite in production, and how to decide which of the two is the copy you keep.

What actually happens when a process logs a line

On any systemd-based distribution, systemd-journald is the first stop for almost everything. It is not one option among several. It sits underneath.

• Anything a unit writes to stdout or stderr, because systemd wires those to the journal by default.
• Kernel messages, read from the kernel ring buffer.
• Classic syslog(3) calls from libc, which land on the /dev/log socket that journald owns.
• Native journal API calls from anything linked against libsystemd, which carry structured key-value fields instead of a flat string.
• Audit records, if Audit=yes is in effect.

journald writes all of that into indexed binary journal files. Then, on most server distributions, rsyslog gets a second copy of the same events and writes them out as text into /var/log/messages, /var/log/secure, /var/log/maillog and friends.

That handoff happens one of two ways, and which one your box uses matters more than most people expect. More on that below. First, the part that costs you an outage.

Before changing anything, look at the effective journald configuration rather than the file you think is authoritative. Distributions ship drop-ins under /usr/lib/systemd/journald.conf.d/ that quietly override upstream defaults:

# Print the merged configuration, including every drop-in, in load order
systemd-analyze cat-config systemd/journald.conf

This is the single most useful command in this whole post. Half the arguments about “what the default is” evaporate once you run it, because upstream systemd and your distribution frequently disagree about ForwardToSyslog.

Failure one: the journal that was never on disk

journald’s Storage= setting defaults to auto. That word does more work than it looks like. Under auto, journald writes to /var/log/journal/ only if that directory already exists. If it does not, journald falls back to /run/log/journal/, which is tmpfs. Memory. Wiped on reboot.

Nothing warns you. journalctl works fine, colours are pretty, filters work. You just silently have no history.

The quickest test is boot history. If the journal is persistent you see multiple boots; if it is volatile you see exactly one:

journalctl --list-boots

To make it persistent, create the directory, let systemd-tmpfiles apply the correct ownership and ACLs, then restart the daemon and flush anything still held in RAM:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush
journalctl --list-boots

The systemd-tmpfiles step is the one people skip. The journal directory needs specific group ownership and ACLs so that unprivileged users can read their own entries and members of the systemd-journal group can read everything. Creating the directory with a bare mkdir and walking away usually works, but it is the kind of “usually” that produces a confusing permissions ticket six months later.

If you want the behaviour to be explicit rather than inferred from a directory’s existence, set it in /etc/systemd/journald.conf.d/00-storage.conf:

[Journal]
Storage=persistent

A drop-in file is better than editing the main config, because package upgrades will not fight you over it.

Failure two: paying for the same log line twice

Once the journal is persistent, you are storing everything twice on a default server build: once as compressed binary in the journal, once as text under /var/log/. Two stores, two completely separate retention policies, neither of which knows the other exists.

The journal is capped by SystemMaxUse= and SystemKeepFree=. Left unset, they default to 10 percent and 15 percent of the filesystem respectively, and journald honours whichever is stricter. Recent systemd also caps those calculated defaults so they do not grow without bound on a large disk, which is worth knowing if you have ever wondered why a huge volume did not produce a proportionally huge journal.

The text copy is capped by logrotate, configured somewhere completely different, usually /etc/logrotate.d/rsyslog, on a weekly or daily schedule with its own rotate count.

Measure both before you tune either:

# What the journal is actually consuming on disk
journalctl --disk-usage

# What everything under /var/log costs, sorted, biggest last
sudo du -sh /var/log/* | sort -h

On a small VPS this stops being academic quickly. A modest instance from a provider like InterServer, Hetzner or DigitalOcean often has /var sharing a single root volume of 20 to 40 GB. A journal allowed to take 10 percent of that, plus four weeks of rotated text logs, plus a database, plus a container image cache, is how you end up paged at two in the morning for something that is not a real incident.

If you decide the text files are the copy you keep, put the journal on a short leash:

[Journal]
# Hard ceiling on total journal size
SystemMaxUse=200M
# Never let the journal push free space below this
SystemKeepFree=1G
# Discard entries older than this regardless of size
MaxRetentionSec=2day

Those three do different jobs and you generally want at least two of them. SystemMaxUse bounds the journal itself. SystemKeepFree protects the rest of the filesystem from the journal. MaxRetentionSec bounds it in time, which is what auditors and data-retention policies actually care about.

To reclaim space immediately without waiting for the next rotation:

# Trim archived journal files until the total falls below 500M
sudo journalctl --vacuum-size=500M

# Or trim by age instead
sudo journalctl --vacuum-time=7d

One caveat that surprises people: vacuuming only removes archived journal files. The currently active file is never deleted. If almost all your usage is in one large active file, run journalctl --rotate first, then vacuum.

Failure three: two rate limiters, both silent

This is the one that is invisible until it matters, and it is the real reason to understand the journald vs rsyslog relationship rather than treating them as interchangeable.

There are two independent rate limiters in the default pipeline, and both drop messages quietly.

journald’s limiter

journald applies RateLimitIntervalSec= and RateLimitBurst= per service, defaulting to 30 seconds and 10000 messages. Exceed the burst inside the interval and the rest of that service’s messages in that window are discarded. Not queued. Discarded.

The one piece of good news is that journald tells you, in the journal itself. Grep for it:

journalctl --grep="Suppressed" -n 50 --no-pager

A chatty web server or a container using the journald log driver will hit this. Rather than disabling rate limiting globally, override it for the one unit that needs it, using a drop-in:

# /etc/systemd/system/nginx.service.d/logging.conf
[Service]
LogRateLimitIntervalSec=30s
LogRateLimitBurst=50000

Per-unit values override journald.conf for that unit only, which is exactly what you want. Turning the global limiter off means one runaway process can fill your disk and take the box down. Turning it up for the single service you know is noisy is a bounded decision.

rsyslog’s limiter

If rsyslog is reading via the imjournal module, it applies its own rate limit on top, defaulting to 20000 messages per 600 seconds. A message can therefore survive journald’s limiter, land in the journal, and still never reach /var/log/messages or your remote log host.

That is the nasty case, because your local investigation finds the events but your centralised alerting never fired. Two people look at two sources and reach opposite conclusions about whether something happened.

rsyslog exposes counters for this through its statistics module, including how many messages it read from the journal, how many it submitted onward, and how many it discarded to rate limiting. Enable impstat if you run anything log-volume-sensitive; the discard counter is the number that tells you whether your pipeline is lying to you.

How rsyslog gets its copy: imjournal or imuxsock

There are two mechanisms, they behave differently under load, and running both at once is a classic source of duplicate lines.

imuxsock: journald pushes a copy to a socket

With ForwardToSyslog=yes in journald, journald writes a classic syslog-formatted copy of each message to a dedicated socket, and rsyslog reads it with imuxsock.

• Where it wins: simple, no state file, no second rate limiter, and no possibility of rsyslog reading back its own output and looping.
• Where it does not: you get the flat syslog view only. Structured journal fields such as the originating unit are not carried across.

imjournal: rsyslog pulls from the journal

imjournal reads journal files directly and keeps a state file so it can resume where it left off after a restart.

• Where it wins: structured fields survive the trip. If you want to route or template on the systemd unit name rather than parsing it out of a message string, this is the only option of the two.
• Where it does not: extra rate limiter to reason about, a state file that can drift, and a documented risk that a corrupted journal database causes rsyslog to re-read the same entries in a loop. rsyslog’s own documentation recommends using imuxsock instead unless you specifically need the structured data.

A representative imjournal load line, with the rate limit stated explicitly rather than left implicit:

module(load="imjournal"
       StateFile="imjournal.state"
       Ratelimit.Interval="600"
       Ratelimit.Burst="20000")

Writing the defaults out by hand is worth the two extra lines. The next person to debug missing messages will see the limiter exists instead of assuming there isn’t one.

To find out which mechanism your box is using right now:

grep -R "imjournal|imuxsock" /etc/rsyslog.conf /etc/rsyslog.d/ 2>/dev/null
grep -R "ForwardToSyslog" /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ /usr/lib/systemd/journald.conf.d/ 2>/dev/null

If the first command returns both modules and the second returns ForwardToSyslog=yes, you have a duplication problem. Pick one path and disable the other.

Only one of the two leaves the box

This is the argument that settles most journald vs rsyslog debates in practice. journald is a local store. In a stock server install it has no mechanism for shipping logs to another host. There are separate systemd components for journal upload and reception, and third-party projects that read the journal and forward it, but none of that is in the default path.

rsyslog forwards as a first-class feature, and it does the part that actually matters: it buffers when the destination is unreachable.

global(workDirectory="/var/spool/rsyslog")

action(type="omfwd"
       target="logs.example.net"
       port="514"
       protocol="tcp"
       action.resumeRetryCount="-1"
       queue.type="LinkedList"
       queue.filename="fwd_loghost"
       queue.maxDiskSpace="1g"
       queue.saveOnShutdown="on")

Line by line, because these are not decoration:

• workDirectory is where spool files get written. Without it, the disk queue has nowhere to go.
• protocol="tcp" rather than UDP. UDP syslog silently drops under congestion, which defeats the point of forwarding at all.
• queue.type="LinkedList" makes the action asynchronous so a slow destination does not block local processing.
• queue.filename is what actually enables disk assistance. In-memory queue first, spilling to disk when it fills. The name must be unique across every action in the config.
• queue.maxDiskSpace caps that spool. Set it, or a long outage at the far end fills the disk you were trying to protect.
• queue.saveOnShutdown="on" persists whatever is still queued across a service restart.

One honest caveat on action.resumeRetryCount="-1". Infinite retry is the right default for “never lose a log line”, but there are reported cases where a relay under sustained back-pressure stops accepting new inbound messages while retrying forever. If your host is a relay rather than a leaf, test that behaviour deliberately with the destination firewalled off before you trust it.

rsyslog is not the only way off the box. Grafana Alloy into Loki, Vector, and Fluent Bit all read the journal directly and speak modern backends, and hosted platforms such as Better Stack, Papertrail or Datadog will take a plain syslog stream if you would rather not run the storage side. The mechanism differs; the decision does not. Something has to own the write path off the machine, and journald on its own is not it.

journald vs rsyslog: how I would decide

Stop asking which is better. Ask which one is the copy you are willing to be judged on, then make the other one cheap.

1. Do logs need to leave this machine? If yes, rsyslog or a dedicated shipper owns the outbound path. That is settled before anything else.
2. Do you troubleshoot mostly by unit? If journalctl -u something -b is your muscle memory, journald is your primary read path and should get the retention budget.
3. Do existing tools read text files? Fail2ban, logwatch, older SIEM collectors and a lot of home-grown scripts parse /var/log/*. Turning rsyslog off breaks them quietly.
4. How much disk do you actually have? Under about 40 GB, keeping two full copies is an unforced error. Cap one hard.
5. Is tamper-evidence a requirement? journald supports Forward Secure Sealing, which detects after-the-fact modification of journal files. Shipping to an append-only remote store is the more commonly accepted answer, and the two are not mutually exclusive.

The configuration I reach for first on a general-purpose server: journald persistent with a hard cap for a few days of fast local querying, rsyslog kept for the syslog files that other tools expect plus reliable forwarding off-box, imuxsock as the handoff unless something genuinely needs structured fields, and logrotate retention trimmed down because the remote copy is the one that matters for anything older than a week.

Troubleshooting

• Journal only shows the current boot. Volatile storage. journalctl --list-boots confirms it, and the fix is the persistent-storage sequence above.
• Events in journalctl but not in /var/log/messages. Either rsyslog is not reading from the journal at all, or the imjournal rate limiter is discarding. Check which input module is loaded first.
• Every line appears twice in the text logs. Both imjournal and imuxsock are active while ForwardToSyslog=yes. Disable one path.
• Bursts of messages disappear during incidents. That is a rate limiter, and incidents are exactly when services get loud. Search for suppression notices and raise the limit for that specific unit.
• journalctl reports corruption or behaves oddly. Run journalctl --verify. If files are damaged, journalctl --rotate starts a fresh active file so new writes are clean, then vacuum the bad archives.
• “Journal has been rotated since unit was started.” Not an error. Rotation happened mid-session, so journalctl -u cannot map the full range. Re-run the query without the unit filter or widen the time range.
• Disk still full after vacuuming. Vacuum skips the active journal file. Rotate first, then vacuum again.

Common mistakes

• Assuming the journal is persistent because journalctl returns results. It always returns results. The question is how far back.
• Disabling rsyslog on a box where fail2ban, logwatch or a scraper still reads /var/log/secure. Nothing errors. Detection just stops.
• Setting RateLimitBurst=0 globally to “stop losing logs”, then having a crash-looping service fill the disk in an afternoon.
• Editing /etc/systemd/journald.conf directly and being surprised when a vendor drop-in overrides it. Use a drop-in of your own with a name that sorts later.
• Forwarding over UDP because it is the one-line version. It drops silently under exactly the load that produced the logs you wanted.
• Enabling a disk-assisted queue without queue.maxDiskSpace, turning a remote outage into a local disk-full outage.
• Tuning journald limits on a machine whose real problem is that one application logs every health check at info level. Fix the source first.

Best practices

• Make storage explicit. Storage=persistent or Storage=volatile, never left to auto on a server you care about.
• Always set SystemKeepFree alongside SystemMaxUse. The first bounds the journal, the second protects everything else on the volume from it.
• Decide consciously which store is authoritative, and write it in the runbook. Two stores with no stated owner means neither gets maintained.
• Use exactly one journal-to-rsyslog path. Both is duplication; neither is silent data loss.
• Override rate limits per unit, not globally.
• Monitor journalctl --disk-usage as a metric, not as something you check after the alert.
• Ship off-box with TCP and a bounded disk-assisted queue, and test it with the destination firewalled off before you rely on it.
• Keep the local copy short and the remote copy long. Local disk is the expensive place to store history.

Frequently asked questions

Can I just disable rsyslog and use journald only?

On a self-contained box with no remote logging and nothing parsing text files, yes, and it saves real disk. Before you do it, grep your configuration management and cron jobs for /var/log/ paths, and check whether fail2ban or any monitoring agent reads those files. That is where this bites people.

Can I disable journald and use rsyslog only?

Not meaningfully. journald is how systemd captures unit stdout and stderr, so it stays in the path regardless. What you can do is set Storage=volatile with a small RuntimeMaxUse so it holds a short in-memory window and rsyslog owns everything durable.

Where are journald logs stored?

/var/log/journal/ when persistent, /run/log/journal/ when volatile. They are indexed binary files, not text, so grep and tail do not work on them directly. Use journalctl, or journalctl -o json if you want to pipe structured output into something else.

Why does journalctl show entries that never reached /var/log/messages?

Three usual causes: rsyslog is not reading the journal at all, the imjournal rate limiter dropped them, or a severity filter such as MaxLevelSyslog or an rsyslog rule excluded that facility or priority. Check in that order.

Is the binary journal format a problem for log analysis?

Locally, no. Structured fields and fast filtering are genuinely better than parsing text with regex. It becomes a problem when you need to move logs somewhere else, because almost every collector expects text or JSON. That conversion step is precisely the role rsyslog or a modern shipper plays.

How much disk should the journal be allowed to use?

Work backwards from how far back you actually query locally. If you rarely look past yesterday, a few hundred megabytes with MaxRetentionSec set to two or three days is plenty. Anything older belongs in a remote store where it is cheaper and survives the machine.

Does journald compress logs automatically?

Yes. Compress= defaults to enabled and applies to entries above a size threshold, which is one reason the journal is often smaller than the equivalent text files despite carrying more metadata. It does not remove the need for a size cap.

The one thing worth remembering

The journald vs rsyslog question is not a contest. On a stock server both are running, both are storing the same events, and the defaults were chosen by your distribution rather than by you. The failure modes all come from that: a journal in RAM that vanishes on reboot, two uncoordinated retention policies eating the same disk, and two rate limiters dropping messages without anyone noticing.

Run systemd-analyze cat-config systemd/journald.conf and journalctl --list-boots on your servers. Two commands, thirty seconds, and you will know whether the logs you would reach for during an incident are actually there.

Need help sorting out your logging pipeline?

Logging is one of those areas where the defaults work well enough to hide the problem until an incident makes it expensive. If any of the above sounded familiar, these are the things I take on:

• Auditing journald and rsyslog on existing servers and telling you exactly where each log line lives, how long it survives, and what is being silently dropped.
• Fixing volatile journals, runaway /var growth and duplicate text-and-binary storage on small VPS instances.
• Setting up reliable off-box forwarding with disk-assisted queues, TCP or RELP transport, and TLS where it is required.
• Migrating from text-file scraping to a structured pipeline into Loki, Elasticsearch or a hosted log platform, without losing the alerts you already depend on.
• Tuning rate limits and retention per service so noisy applications stop hiding the messages that matter.
• Writing the runbook that says which store is authoritative, so the next engineer does not have to reverse-engineer it during an outage.

Send me your journald.conf, your rsyslog.conf, or the output of journalctl --disk-usage and I will tell you what I see before we talk about scope.

The post journald vs rsyslog: Where Your Linux Logs Actually Live appeared first on John Nessime.