Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker’s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.

This post covers Docker log rotation properly: what the defaults actually do, why the daemon.json change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.

The default is unbounded, and that is documented behaviour

Docker’s default logging driver is json-file. It captures stdout and stderr from the container’s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.

The important part is the option defaults. For json-file, max-size defaults to unlimited, max-file defaults to 1, and compress defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.

Docker’s own documentation is upfront about why. Keeping json-file unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.

Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:

# What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"

The gap between the first command and the second is where most of the pain lives, and we will come back to it.

To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container’s log file plus a small amount of metadata, so the sizes are close enough to be useful:

sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10

Container IDs are not memorable, so map the winner back to a name with docker ps --no-trunc or by grepping the ID against docker ps -aq. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.

Setting up Docker log rotation on the daemon

The daemon-wide fix goes in /etc/docker/daemon.json. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

What each key buys you:

• max-size caps a single file. Accepts an integer plus k, m or g. This is the setting that stops the runaway.
• max-file caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is only effective if max-size is also set, which is the single most common misconfiguration I see.
• compress gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.

Two things trip people up here. First, every value in log-opts must be a JSON string, including the numeric ones. Writing "max-file": 3 without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed daemon.json on a production host is a much worse afternoon than a full disk:

# Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  && echo "daemon.json parses OK"

sudo systemctl restart docker

Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is max-size multiplied by max-file. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.

Why your daemon.json change did nothing

This is the failure mode that costs the most time, because it looks exactly like the fix not working.

Logging configuration is baked into a container’s host config when the container is created. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because docker restart stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.

So the container that filled your disk keeps filling your disk, its log file still has no cap, and docker info cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.

To actually apply it, recreate the container and then verify rather than assuming:

# Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web

You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.

Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.

Pinning limits per service in Compose

Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides daemon.json entirely. A YAML anchor keeps it from turning into copy-paste sprawl:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

Be aware that the logging block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with its defaults, which for json-file means straight back to unbounded. Half-specifying is worse than not specifying at all.

You deleted the log file and df did not move

Classic 3am mistake, and it is worth understanding rather than memorising.

When you rm the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from ls, the daemon keeps writing into a file you can no longer see, and df reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.

truncate -s 0 is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:

sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"

There is a cost. Truncating out from under the daemon can leave an in-flight docker logs -f stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.

The same reasoning applies to logrotate rules pointed at /var/lib/docker/containers/*/*.log. Rotating with copytruncate works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker’s own docs warn against external tools touching those files. If the driver can rotate for you, let it.

Choosing a driver, and what each one costs

json-file

The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. docker logs works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.

local

Docker’s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it rotates and compresses by default, retaining five files of 20 MB each per container out of the box. docker logs works exactly as normal. The trade-off is that the format is Docker’s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is docker logs plus a shipper that talks to the daemon, this is the safest default you can pick.

journald

Hands container output to systemd’s journal, which means you get journalctl filtering, structured fields, and the journal’s own size management via SystemMaxUse in journald.conf. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.

The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at RateLimitIntervalSec and RateLimitBurst before you need them, not after.

syslog, fluentd and the remote drivers

These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.

What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.

none

Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because docker logs returns nothing and you will waste twenty minutes concluding the container is broken.

The blocking behaviour worth knowing about

By default the logging path is blocking. If the driver cannot keep up, the container’s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.

docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1

Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.

Dual logging, and why docker logs still works

If you switch to a remote driver and find docker logs still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the local driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.

Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.

Reclaiming space when the disk is already full

Order matters here. Free space first, fix the cause second, verify third.

1. Confirm logs are actually the problem. Compare du -sh /var/lib/docker/containers against docker system df, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.
2. Identify the largest log files with the du and sort command from earlier.
3. Copy anything you need out first, if the logs are part of an active investigation.
4. Truncate the offenders with truncate -s 0. Never rm.
5. Set the limits in daemon.json, validate the JSON, restart the daemon.
6. Recreate the containers so the limits take effect, then confirm with docker inspect on each one.
7. Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.

Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.

Troubleshooting

Rotation is configured but files still grow past the cap. The container predates the change. Check docker inspect --format '{{.HostConfig.LogConfig}}' and recreate it.

The daemon will not start after editing daemon.json. Almost always invalid JSON, or unquoted values in log-opts. Read the actual error with journalctl -u docker --no-pager -n 50 rather than guessing.

max-file is set but only one file ever appears. max-file does nothing without max-size. Set both.

docker logs -f hangs after showing existing output. Something truncated the file underneath the daemon, usually an external logrotate rule. Reattach to recover, then remove the rule.

docker logs returns nothing at all. Check the driver. none discards everything, and some remote drivers only serve reads through the dual-logging cache.

Disk usage did not drop after deleting log files. The daemon still holds the descriptors. Restart the daemon to release the inodes, and use truncate next time.

Logs vanish sooner than the retention you configured. On journald, check journal rate limiting and SystemMaxUse. On a remote driver in non-blocking mode, check whether the buffer is overflowing.

Common mistakes

• Assuming a daemon restart applies the new limits to running containers. It does not.
• Setting max-file without max-size and believing rotation is on.
• Writing numeric values unquoted in log-opts.
• Overwriting an existing daemon.json instead of merging, and silently dropping storage or network settings.
• Using rm on a live container’s log file and concluding the disk report is broken.
• Specifying a logging.driver in Compose without its options, which resets that service back to the driver’s own unbounded defaults.
• Pointing host logrotate at the Docker containers directory when the driver could have handled it.
• Sizing rotation per container without multiplying by container count against the actual disk.

Best practices

• Set logging limits in daemon.json as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.
• Prefer the local driver on standalone hosts unless something specifically needs to read the JSON files.
• Turn on compress. Log text compresses well enough that you get more retention for the same disk.
• Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.
• Do the multiplication. Worst-case host usage is max-size × max-file × container count, and it should be a fraction of your disk, not most of it.
• Alert on Docker data root usage at a threshold that leaves you time to act.
• Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.
• Verify with docker inspect after any change. docker info tells you the daemon’s intent, not what your containers carry.

Frequently asked questions

Does Docker rotate container logs automatically?

Not with the default driver. json-file ships with max-size unlimited and max-file set to 1, so a single file grows without bound. The local driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.

Where are Docker container logs stored on the host?

Under the Docker data root, one directory per container, which on a standard Linux install means /var/lib/docker/containers/. Rather than assembling the path by hand, ask the daemon: docker inspect --format '{{.LogPath}}' <container>. That works regardless of a custom data-root.

How do I clear Docker logs without stopping the container?

Truncate the file to zero bytes with truncate -s 0 on the path from {{.LogPath}}. Do not use rm: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.

Why did my daemon.json rotation settings not apply?

Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and docker restart reuses the same container object. Recreate the container, then confirm with docker inspect --format '{{.HostConfig.LogConfig}}'.

Should I use the local driver or json-file?

Use local if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use json-file if an agent or shipper on the host tails those files, because it cannot parse the local format. Either way docker logs behaves the same.

Can I use logrotate for Docker container logs instead?

You can, and plenty of hosts do, but the driver options are the better tool. A copytruncate rule pointed at the containers directory reaches into files the daemon treats as private, and can leave docker logs -f streams stalled after each rotation. Use logrotate for logs your applications write to mounted volumes, and let the driver handle stdout.

How much disk should I budget for container logs?

Worst case is max-size × max-file per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.

The one thing worth remembering

Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.

So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with docker inspect rather than trusting docker info. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.

Need a hand with container logging on your hosts?

Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:

• Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits
• Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number
• Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back
• Choosing between json-file, local, journald and shipping off-box, including the blocking behaviour trade-offs under load
• Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends
• Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot

If you want a second opinion, send me your daemon.json, the output of docker ps and a du -sh of your containers directory, and I will tell you what I would change.

The post Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation appeared first on John Nessime.

The worse version of this problem is the mirror image. You reboot a server to clear something, come back to read what led up to it, and the journal starts at the reboot. Everything before is gone. Not rotated, not compressed, not archived somewhere clever. Gone, because the journal was living in RAM and nobody ever told it otherwise.

Both of those are the same misunderstanding wearing different clothes: not knowing where your logs actually live. This post walks through the journald vs rsyslog split on a modern Linux server, what each one really stores, the three failure modes that bite in production, and how to decide which of the two is the copy you keep.

What actually happens when a process logs a line

On any systemd-based distribution, systemd-journald is the first stop for almost everything. It is not one option among several. It sits underneath.

• Anything a unit writes to stdout or stderr, because systemd wires those to the journal by default.
• Kernel messages, read from the kernel ring buffer.
• Classic syslog(3) calls from libc, which land on the /dev/log socket that journald owns.
• Native journal API calls from anything linked against libsystemd, which carry structured key-value fields instead of a flat string.
• Audit records, if Audit=yes is in effect.

journald writes all of that into indexed binary journal files. Then, on most server distributions, rsyslog gets a second copy of the same events and writes them out as text into /var/log/messages, /var/log/secure, /var/log/maillog and friends.

That handoff happens one of two ways, and which one your box uses matters more than most people expect. More on that below. First, the part that costs you an outage.

Before changing anything, look at the effective journald configuration rather than the file you think is authoritative. Distributions ship drop-ins under /usr/lib/systemd/journald.conf.d/ that quietly override upstream defaults:

# Print the merged configuration, including every drop-in, in load order
systemd-analyze cat-config systemd/journald.conf

This is the single most useful command in this whole post. Half the arguments about “what the default is” evaporate once you run it, because upstream systemd and your distribution frequently disagree about ForwardToSyslog.

Failure one: the journal that was never on disk

journald’s Storage= setting defaults to auto. That word does more work than it looks like. Under auto, journald writes to /var/log/journal/ only if that directory already exists. If it does not, journald falls back to /run/log/journal/, which is tmpfs. Memory. Wiped on reboot.

Nothing warns you. journalctl works fine, colours are pretty, filters work. You just silently have no history.

The quickest test is boot history. If the journal is persistent you see multiple boots; if it is volatile you see exactly one:

journalctl --list-boots

To make it persistent, create the directory, let systemd-tmpfiles apply the correct ownership and ACLs, then restart the daemon and flush anything still held in RAM:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush
journalctl --list-boots

The systemd-tmpfiles step is the one people skip. The journal directory needs specific group ownership and ACLs so that unprivileged users can read their own entries and members of the systemd-journal group can read everything. Creating the directory with a bare mkdir and walking away usually works, but it is the kind of “usually” that produces a confusing permissions ticket six months later.

If you want the behaviour to be explicit rather than inferred from a directory’s existence, set it in /etc/systemd/journald.conf.d/00-storage.conf:

[Journal]
Storage=persistent

A drop-in file is better than editing the main config, because package upgrades will not fight you over it.

Failure two: paying for the same log line twice

Once the journal is persistent, you are storing everything twice on a default server build: once as compressed binary in the journal, once as text under /var/log/. Two stores, two completely separate retention policies, neither of which knows the other exists.

The journal is capped by SystemMaxUse= and SystemKeepFree=. Left unset, they default to 10 percent and 15 percent of the filesystem respectively, and journald honours whichever is stricter. Recent systemd also caps those calculated defaults so they do not grow without bound on a large disk, which is worth knowing if you have ever wondered why a huge volume did not produce a proportionally huge journal.

The text copy is capped by logrotate, configured somewhere completely different, usually /etc/logrotate.d/rsyslog, on a weekly or daily schedule with its own rotate count.

Measure both before you tune either:

# What the journal is actually consuming on disk
journalctl --disk-usage

# What everything under /var/log costs, sorted, biggest last
sudo du -sh /var/log/* | sort -h

On a small VPS this stops being academic quickly. A modest instance from a provider like InterServer, Hetzner or DigitalOcean often has /var sharing a single root volume of 20 to 40 GB. A journal allowed to take 10 percent of that, plus four weeks of rotated text logs, plus a database, plus a container image cache, is how you end up paged at two in the morning for something that is not a real incident.

If you decide the text files are the copy you keep, put the journal on a short leash:

[Journal]
# Hard ceiling on total journal size
SystemMaxUse=200M
# Never let the journal push free space below this
SystemKeepFree=1G
# Discard entries older than this regardless of size
MaxRetentionSec=2day

Those three do different jobs and you generally want at least two of them. SystemMaxUse bounds the journal itself. SystemKeepFree protects the rest of the filesystem from the journal. MaxRetentionSec bounds it in time, which is what auditors and data-retention policies actually care about.

To reclaim space immediately without waiting for the next rotation:

# Trim archived journal files until the total falls below 500M
sudo journalctl --vacuum-size=500M

# Or trim by age instead
sudo journalctl --vacuum-time=7d

One caveat that surprises people: vacuuming only removes archived journal files. The currently active file is never deleted. If almost all your usage is in one large active file, run journalctl --rotate first, then vacuum.

Failure three: two rate limiters, both silent

This is the one that is invisible until it matters, and it is the real reason to understand the journald vs rsyslog relationship rather than treating them as interchangeable.

There are two independent rate limiters in the default pipeline, and both drop messages quietly.

journald’s limiter

journald applies RateLimitIntervalSec= and RateLimitBurst= per service, defaulting to 30 seconds and 10000 messages. Exceed the burst inside the interval and the rest of that service’s messages in that window are discarded. Not queued. Discarded.

The one piece of good news is that journald tells you, in the journal itself. Grep for it:

journalctl --grep="Suppressed" -n 50 --no-pager

A chatty web server or a container using the journald log driver will hit this. Rather than disabling rate limiting globally, override it for the one unit that needs it, using a drop-in:

# /etc/systemd/system/nginx.service.d/logging.conf
[Service]
LogRateLimitIntervalSec=30s
LogRateLimitBurst=50000

Per-unit values override journald.conf for that unit only, which is exactly what you want. Turning the global limiter off means one runaway process can fill your disk and take the box down. Turning it up for the single service you know is noisy is a bounded decision.

rsyslog’s limiter

If rsyslog is reading via the imjournal module, it applies its own rate limit on top, defaulting to 20000 messages per 600 seconds. A message can therefore survive journald’s limiter, land in the journal, and still never reach /var/log/messages or your remote log host.

That is the nasty case, because your local investigation finds the events but your centralised alerting never fired. Two people look at two sources and reach opposite conclusions about whether something happened.

rsyslog exposes counters for this through its statistics module, including how many messages it read from the journal, how many it submitted onward, and how many it discarded to rate limiting. Enable impstat if you run anything log-volume-sensitive; the discard counter is the number that tells you whether your pipeline is lying to you.

How rsyslog gets its copy: imjournal or imuxsock

There are two mechanisms, they behave differently under load, and running both at once is a classic source of duplicate lines.

imuxsock: journald pushes a copy to a socket

With ForwardToSyslog=yes in journald, journald writes a classic syslog-formatted copy of each message to a dedicated socket, and rsyslog reads it with imuxsock.

• Where it wins: simple, no state file, no second rate limiter, and no possibility of rsyslog reading back its own output and looping.
• Where it does not: you get the flat syslog view only. Structured journal fields such as the originating unit are not carried across.

imjournal: rsyslog pulls from the journal

imjournal reads journal files directly and keeps a state file so it can resume where it left off after a restart.

• Where it wins: structured fields survive the trip. If you want to route or template on the systemd unit name rather than parsing it out of a message string, this is the only option of the two.
• Where it does not: extra rate limiter to reason about, a state file that can drift, and a documented risk that a corrupted journal database causes rsyslog to re-read the same entries in a loop. rsyslog’s own documentation recommends using imuxsock instead unless you specifically need the structured data.

A representative imjournal load line, with the rate limit stated explicitly rather than left implicit:

module(load="imjournal"
       StateFile="imjournal.state"
       Ratelimit.Interval="600"
       Ratelimit.Burst="20000")

Writing the defaults out by hand is worth the two extra lines. The next person to debug missing messages will see the limiter exists instead of assuming there isn’t one.

To find out which mechanism your box is using right now:

grep -R "imjournal|imuxsock" /etc/rsyslog.conf /etc/rsyslog.d/ 2>/dev/null
grep -R "ForwardToSyslog" /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ /usr/lib/systemd/journald.conf.d/ 2>/dev/null

If the first command returns both modules and the second returns ForwardToSyslog=yes, you have a duplication problem. Pick one path and disable the other.

Only one of the two leaves the box

This is the argument that settles most journald vs rsyslog debates in practice. journald is a local store. In a stock server install it has no mechanism for shipping logs to another host. There are separate systemd components for journal upload and reception, and third-party projects that read the journal and forward it, but none of that is in the default path.

rsyslog forwards as a first-class feature, and it does the part that actually matters: it buffers when the destination is unreachable.

global(workDirectory="/var/spool/rsyslog")

action(type="omfwd"
       target="logs.example.net"
       port="514"
       protocol="tcp"
       action.resumeRetryCount="-1"
       queue.type="LinkedList"
       queue.filename="fwd_loghost"
       queue.maxDiskSpace="1g"
       queue.saveOnShutdown="on")

Line by line, because these are not decoration:

• workDirectory is where spool files get written. Without it, the disk queue has nowhere to go.
• protocol="tcp" rather than UDP. UDP syslog silently drops under congestion, which defeats the point of forwarding at all.
• queue.type="LinkedList" makes the action asynchronous so a slow destination does not block local processing.
• queue.filename is what actually enables disk assistance. In-memory queue first, spilling to disk when it fills. The name must be unique across every action in the config.
• queue.maxDiskSpace caps that spool. Set it, or a long outage at the far end fills the disk you were trying to protect.
• queue.saveOnShutdown="on" persists whatever is still queued across a service restart.

One honest caveat on action.resumeRetryCount="-1". Infinite retry is the right default for “never lose a log line”, but there are reported cases where a relay under sustained back-pressure stops accepting new inbound messages while retrying forever. If your host is a relay rather than a leaf, test that behaviour deliberately with the destination firewalled off before you trust it.

rsyslog is not the only way off the box. Grafana Alloy into Loki, Vector, and Fluent Bit all read the journal directly and speak modern backends, and hosted platforms such as Better Stack, Papertrail or Datadog will take a plain syslog stream if you would rather not run the storage side. The mechanism differs; the decision does not. Something has to own the write path off the machine, and journald on its own is not it.

journald vs rsyslog: how I would decide

Stop asking which is better. Ask which one is the copy you are willing to be judged on, then make the other one cheap.

1. Do logs need to leave this machine? If yes, rsyslog or a dedicated shipper owns the outbound path. That is settled before anything else.
2. Do you troubleshoot mostly by unit? If journalctl -u something -b is your muscle memory, journald is your primary read path and should get the retention budget.
3. Do existing tools read text files? Fail2ban, logwatch, older SIEM collectors and a lot of home-grown scripts parse /var/log/*. Turning rsyslog off breaks them quietly.
4. How much disk do you actually have? Under about 40 GB, keeping two full copies is an unforced error. Cap one hard.
5. Is tamper-evidence a requirement? journald supports Forward Secure Sealing, which detects after-the-fact modification of journal files. Shipping to an append-only remote store is the more commonly accepted answer, and the two are not mutually exclusive.

The configuration I reach for first on a general-purpose server: journald persistent with a hard cap for a few days of fast local querying, rsyslog kept for the syslog files that other tools expect plus reliable forwarding off-box, imuxsock as the handoff unless something genuinely needs structured fields, and logrotate retention trimmed down because the remote copy is the one that matters for anything older than a week.

Troubleshooting

• Journal only shows the current boot. Volatile storage. journalctl --list-boots confirms it, and the fix is the persistent-storage sequence above.
• Events in journalctl but not in /var/log/messages. Either rsyslog is not reading from the journal at all, or the imjournal rate limiter is discarding. Check which input module is loaded first.
• Every line appears twice in the text logs. Both imjournal and imuxsock are active while ForwardToSyslog=yes. Disable one path.
• Bursts of messages disappear during incidents. That is a rate limiter, and incidents are exactly when services get loud. Search for suppression notices and raise the limit for that specific unit.
• journalctl reports corruption or behaves oddly. Run journalctl --verify. If files are damaged, journalctl --rotate starts a fresh active file so new writes are clean, then vacuum the bad archives.
• “Journal has been rotated since unit was started.” Not an error. Rotation happened mid-session, so journalctl -u cannot map the full range. Re-run the query without the unit filter or widen the time range.
• Disk still full after vacuuming. Vacuum skips the active journal file. Rotate first, then vacuum again.

Common mistakes

• Assuming the journal is persistent because journalctl returns results. It always returns results. The question is how far back.
• Disabling rsyslog on a box where fail2ban, logwatch or a scraper still reads /var/log/secure. Nothing errors. Detection just stops.
• Setting RateLimitBurst=0 globally to “stop losing logs”, then having a crash-looping service fill the disk in an afternoon.
• Editing /etc/systemd/journald.conf directly and being surprised when a vendor drop-in overrides it. Use a drop-in of your own with a name that sorts later.
• Forwarding over UDP because it is the one-line version. It drops silently under exactly the load that produced the logs you wanted.
• Enabling a disk-assisted queue without queue.maxDiskSpace, turning a remote outage into a local disk-full outage.
• Tuning journald limits on a machine whose real problem is that one application logs every health check at info level. Fix the source first.

Best practices

• Make storage explicit. Storage=persistent or Storage=volatile, never left to auto on a server you care about.
• Always set SystemKeepFree alongside SystemMaxUse. The first bounds the journal, the second protects everything else on the volume from it.
• Decide consciously which store is authoritative, and write it in the runbook. Two stores with no stated owner means neither gets maintained.
• Use exactly one journal-to-rsyslog path. Both is duplication; neither is silent data loss.
• Override rate limits per unit, not globally.
• Monitor journalctl --disk-usage as a metric, not as something you check after the alert.
• Ship off-box with TCP and a bounded disk-assisted queue, and test it with the destination firewalled off before you rely on it.
• Keep the local copy short and the remote copy long. Local disk is the expensive place to store history.

Frequently asked questions

Can I just disable rsyslog and use journald only?

On a self-contained box with no remote logging and nothing parsing text files, yes, and it saves real disk. Before you do it, grep your configuration management and cron jobs for /var/log/ paths, and check whether fail2ban or any monitoring agent reads those files. That is where this bites people.

Can I disable journald and use rsyslog only?

Not meaningfully. journald is how systemd captures unit stdout and stderr, so it stays in the path regardless. What you can do is set Storage=volatile with a small RuntimeMaxUse so it holds a short in-memory window and rsyslog owns everything durable.

Where are journald logs stored?

/var/log/journal/ when persistent, /run/log/journal/ when volatile. They are indexed binary files, not text, so grep and tail do not work on them directly. Use journalctl, or journalctl -o json if you want to pipe structured output into something else.

Why does journalctl show entries that never reached /var/log/messages?

Three usual causes: rsyslog is not reading the journal at all, the imjournal rate limiter dropped them, or a severity filter such as MaxLevelSyslog or an rsyslog rule excluded that facility or priority. Check in that order.

Is the binary journal format a problem for log analysis?

Locally, no. Structured fields and fast filtering are genuinely better than parsing text with regex. It becomes a problem when you need to move logs somewhere else, because almost every collector expects text or JSON. That conversion step is precisely the role rsyslog or a modern shipper plays.

How much disk should the journal be allowed to use?

Work backwards from how far back you actually query locally. If you rarely look past yesterday, a few hundred megabytes with MaxRetentionSec set to two or three days is plenty. Anything older belongs in a remote store where it is cheaper and survives the machine.

Does journald compress logs automatically?

Yes. Compress= defaults to enabled and applies to entries above a size threshold, which is one reason the journal is often smaller than the equivalent text files despite carrying more metadata. It does not remove the need for a size cap.

The one thing worth remembering

The journald vs rsyslog question is not a contest. On a stock server both are running, both are storing the same events, and the defaults were chosen by your distribution rather than by you. The failure modes all come from that: a journal in RAM that vanishes on reboot, two uncoordinated retention policies eating the same disk, and two rate limiters dropping messages without anyone noticing.

Run systemd-analyze cat-config systemd/journald.conf and journalctl --list-boots on your servers. Two commands, thirty seconds, and you will know whether the logs you would reach for during an incident are actually there.

Need help sorting out your logging pipeline?

Logging is one of those areas where the defaults work well enough to hide the problem until an incident makes it expensive. If any of the above sounded familiar, these are the things I take on:

• Auditing journald and rsyslog on existing servers and telling you exactly where each log line lives, how long it survives, and what is being silently dropped.
• Fixing volatile journals, runaway /var growth and duplicate text-and-binary storage on small VPS instances.
• Setting up reliable off-box forwarding with disk-assisted queues, TCP or RELP transport, and TLS where it is required.
• Migrating from text-file scraping to a structured pipeline into Loki, Elasticsearch or a hosted log platform, without losing the alerts you already depend on.
• Tuning rate limits and retention per service so noisy applications stop hiding the messages that matter.
• Writing the runbook that says which store is authoritative, so the next engineer does not have to reverse-engineer it during an outage.

Send me your journald.conf, your rsyslog.conf, or the output of journalctl --disk-usage and I will tell you what I see before we talk about scope.

The post journald vs rsyslog: Where Your Linux Logs Actually Live appeared first on John Nessime.

Nothing about the Loki config changed. What changed is that three weeks ago somebody added request_id as a label because it seemed helpful, and Loki has been quietly building millions of separate streams ever since. The index that used to fit comfortably in memory does not. Every query now has to open thousands of tiny chunks instead of a handful of large ones.

This is the failure mode that defines centralized logging with Loki, and it is almost entirely self-inflicted. Loki is cheap and fast when you use it the way it was designed and miserable when you bring habits from Elasticsearch. Nothing warns you at the moment you make the mistake. The bill comes weeks later, during an incident.

This covers the model you have to internalise, how to choose labels, what replaced Promtail (it is gone, and every tutorial older than this year is wrong about it), the LogQL patterns worth knowing, and how to keep retention and storage costs under control.

Loki is not Elasticsearch, and that is the entire design

Elasticsearch builds a full inverted index over every token in every log line. Powerful, and the index often ends up as large as the logs themselves. That is where the cost comes from.

Loki indexes only labels. The log content is compressed into chunks and pushed to object storage, untouched and unindexed. A query does two things: it uses the index to select which streams to read, then it brute-force scans the contents of those chunks in parallel.

The consequence is a rule worth writing on a wall: labels select, filters search. Labels exist to narrow down which chunks get opened. Everything else is a filter applied at query time. Scanning is genuinely fast because it is embarrassingly parallel, so filtering on content is not the expensive operation people assume.

What is expensive is streams. A stream is one unique combination of label values, and each gets its own chunks. Do the arithmetic before you add a label:

• 8 jobs, 3 environments, 12 hosts gives you 288 streams. Comfortable.
• Add request_id, and you multiply that by however many requests you serve. The index explodes, chunks get flushed while nearly empty, and object storage fills with millions of tiny files that queries must open individually.

Ingesters also hold every active stream in memory. Cardinality does not just make queries slow; it is how Loki gets OOM-killed.

Choosing labels

A label earns its place if it satisfies all three: the set of values is bounded and small, you know the values in advance, and you would actually use it to narrow a search.

• Good: job, service, env, host, namespace, level.
• Bad: request_id, trace_id, user_id, session_id, full URL paths, IP addresses, timestamps, durations, anything derived from the log content itself.

Two subtler traps. instance including an ephemeral port is unbounded. And in container environments, pod names churn constantly, so labelling by pod name gives you a new stream every deployment forever.

Aim to keep the total under a few thousand active streams. Check rather than guess, using logcli, which reports cardinality per label:

# Which of your labels is quietly out of control?
logcli series '{}' --analyze-labels

Run that before you have a problem. It is the single most useful diagnostic here and almost nobody knows about it.

The escape hatch: structured metadata

“Never label a trace ID” is correct but unsatisfying, because sometimes you genuinely need to find every line for one trace. Loki 3 answers this with structured metadata: key-value pairs attached to a log line, stored alongside it, not in the index and not creating streams.

limits_config:
  # Off by default on older configs. Turn it on before your
  # agent starts sending structured metadata, or the writes
  # get rejected and you spend an hour puzzling over it.
  allow_structured_metadata: true

Query it with the same label-filter syntax, and it costs you nothing in cardinality:

{job="api", env="production"} | trace_id="9f2c1b4e"

So the decision becomes three-way rather than two-way: bounded and used for selection becomes a label, high-cardinality but frequently searched becomes structured metadata, and everything else stays in the line and gets parsed at query time.

Promtail is gone. Use Alloy.

This one has a date and it matters, because it invalidates most of the Loki material on the internet.

Promtail, the agent every Loki tutorial tells you to install, was deprecated in February 2025 and reached end of life on 2 March 2026. No security patches, no bug fixes, no updates. Grafana Alloy, their distribution of the OpenTelemetry Collector, is the replacement, and it handles logs, metrics, traces and profiles in one binary instead of running a separate agent per signal.

If you already run Promtail, there is a conversion tool rather than a rewrite:

# Converts an existing Promtail config to Alloy syntax.
# Read the output before deploying it; the converter flags
# anything it could not translate cleanly.
alloy convert --source-format=promtail --output=config.alloy promtail.yaml

Alloy configuration is a component graph rather than a YAML pipeline. Components reference each other’s outputs, which reads oddly at first and then makes more sense than nested YAML stages:

// Find the files, and attach the labels that will select them later.
// Note what is NOT here: nothing derived from log content.
local.file_match "system" {
  path_targets = [{
    __path__ = "/var/log/syslog",
    job      = "syslog",
    host     = constants.hostname,
    env      = "production",
  }]
}

loki.source.file "system" {
  targets    = local.file_match.system.targets
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Between source and write you can insert a loki.process component to parse lines, drop noise, and promote fields into structured metadata. Do the dropping here rather than at the Loki end: logs you never ship cost nothing to store, index or query, and debug-level chatter from a busy service is usually the single largest slice of ingest.

For Docker hosts, Alloy has components that discover containers and read their logs directly, which is tidier than tailing the JSON files under /var/lib/docker and surviving a logging-driver change.

Deployment shape and storage

Loki ships as one binary that runs in different modes depending on the -target flag.

• Monolithic. Everything in one process. Right for a single VPS, a homelab, or anything up to a modest daily volume. Do not feel bad about this; a lot of people run microservices mode for a workload one process would handle.
• Simple scalable. Splits into read, write and backend targets behind a load balancer. The sensible next step, and where most self-hosted setups should stop.
• Microservices. Every component separately. Genuinely necessary at large scale and a substantial operational burden below it.

For storage, use TSDB with schema v13. Older guides show boltdb-shipper; that is the previous generation. Chunks belong in object storage in anything you care about, because filesystem storage means one disk, one node, and no redundancy.

schema_config:
  configs:
    # Schema changes are additive. Add a new entry with a future
    # 'from' date rather than editing the existing one, or Loki
    # cannot read the data it already wrote.
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

That “additive” note is worth taking seriously. Editing an existing schema entry in place is one of the few ways to make historical logs unreadable.

Any S3-compatible store works, which is where the cost advantage lives. Chunks are compressed and write-once, so cheaper tiers suit them well: S3 itself, or MinIO if you want it self-hosted, or one of the cheaper compatible providers like Backblaze B2 or Cloudflare R2. Watch egress pricing rather than storage pricing, since queries read chunks back.

LogQL worth actually knowing

Every query starts with a stream selector in braces. That part is mandatory and it is what decides how much data gets read.

# Order matters for performance:
#   1. select streams (index lookup)
#   2. filter lines (cheap string match, runs on every line)
#   3. parse (expensive, so run it on the smallest set possible)
{job="api", env="production"} |= "timeout" | json | status="500"

People routinely write | json before the line filter and then wonder why the query is slow. Parsing every line in the range to discard most of them is exactly the wrong order.

Loki also turns logs into metrics, which is where it earns its place next to Prometheus:

# Error rate per service, graphable and alertable.
sum by (job) (rate({env="production"} |= "level=error" [5m]))

# Which streams are eating your ingest budget?
topk(10, sum by (job, host) (count_over_time({env="production"}[1h])))

That second one is worth saving as a dashboard panel. Ingest problems announce themselves there long before they show up as a slow query.

Loki’s ruler evaluates the same expressions as alerting rules and sends them to Alertmanager, so “alert when this service logs more than N errors in five minutes” needs no extra component if you already run Prometheus.

Retention, and why it silently does nothing

Setting retention_period is not enough. Retention is enforced by the compactor, and the compactor does not do it unless you tell it to. Plenty of people set a retention period, watch their bucket keep growing, and never connect the two.

compactor:
  working_directory: /loki/compactor
  # Without this, the compactor compacts the index and
  # deletes nothing. This is the line people miss.
  retention_enabled: true
  delete_request_store: s3

limits_config:
  retention_period: 720h        # 30 days, global default

Per-stream retention lets you keep what matters and discard what does not: audit logs for a year, application debug logs for three days. That difference tends to matter more to the bill than any storage tier choice.

Deletion is asynchronous. Chunks disappear from object storage some time after the index entries go, so do not panic if the bucket does not shrink immediately.

Troubleshooting

Queries are slow or time out

Cardinality first. Run logcli series '{}' --analyze-labels and look for a label with a suspiciously large value count. Then check query shape: a selector that matches everything, a parser before a line filter, or a time range far wider than needed. Adding query frontend caching helps repeated queries but will not rescue a bad label set.

Stream limit exceeded

Loki is telling you the truth. Raising max_global_streams_per_user buys time and makes the underlying problem worse. Find the label that is generating streams and move it to structured metadata or into the line.

Rate limit or ingestion errors

Two different limits produce similar-looking rejections: a global ingestion rate, and a per-stream rate. Per-stream limits usually mean one very chatty stream rather than too much traffic overall, and the fix is to split it by a label or drop the noise at the agent.

Logs are not appearing at all

Work forward from the agent. Check the agent’s own logs first, then whether it can reach the push endpoint, then whether the file is actually readable by the agent’s user. Container log permissions and SELinux catch people constantly. In Grafana, remember the label browser only shows labels seen recently, so an empty dropdown often means nothing has arrived, not that the query is wrong.

Logs stop after an agent restart

The position file that tracks how far it read is not persisting. In a container, that means it is not on a mounted volume, so every restart re-reads or skips depending on configuration.

Storage keeps growing despite retention

Confirm retention_enabled: true is actually set, that exactly one compactor is running, and that it has write access to the object store. A compactor that cannot delete usually says so in its logs and nowhere else.

Common mistakes

• Treating labels like Elasticsearch fields and labelling everything.
• Promoting a parsed value into a label without checking how many distinct values it has.
• Following a tutorial that installs Promtail, which is now end of life.
• Writing | json before the line filter.
• Setting retention_period without retention_enabled: true.
• Running more than one compactor.
• Editing an existing schema_config entry instead of adding a new one.
• Filesystem storage on a single node for anything you would miss.
• Shipping every debug line and then paying to store, index and scan it.
• Raising stream limits instead of fixing cardinality.
• Running microservices mode for a workload the single binary handles comfortably.
• No dashboard showing ingest volume per stream, so growth is invisible until queries break.

Best practices

• Decide labels deliberately: bounded, known in advance, used for selection.
• Put high-cardinality but searchable values in structured metadata, not labels.
• Run --analyze-labels periodically, not just when something breaks.
• Drop noise at the agent, before it costs you anything downstream.
• Emit structured logs from your applications so parsers are cheap and reliable.
• Start monolithic and move to simple scalable only when you have a reason.
• TSDB, schema v13, chunks in object storage.
• Turn retention on properly and use per-stream retention to keep costs sane.
• Order queries: select, filter, parse.
• Alert on log-derived metrics through the ruler rather than eyeballing dashboards.
• Graph ingest volume per job so cardinality growth shows up as a trend.

FAQ

Is Loki a drop-in replacement for the ELK stack?

No, and expecting it to be is where most disappointment comes from. Loki is dramatically cheaper to run and much better suited to “show me the logs for this service around this time”. Elasticsearch is better at ad-hoc full-text search across everything with no idea where to start. Pick based on which of those you do more often.

Why is my Loki so slow?

Almost always label cardinality. Run logcli series '{}' --analyze-labels and look for the outlier. The second most common cause is a query that parses before it filters. Hardware is a distant third.

Should I still use Promtail?

No. It reached end of life in March 2026 and receives no security patches. Use Grafana Alloy, and run alloy convert against your existing config rather than rewriting it from scratch.

How many labels is too many?

The count matters less than the product of their distinct values. Five labels with a handful of values each is fine. Three labels where one has fifty thousand values is not. Multiply before you add.

Can I run Loki on a single VPS?

Yes, and for a handful of services it is a perfectly good answer. Monolithic mode alongside Grafana and Prometheus fits on a modest box from any provider. Point chunk storage at an object store rather than local disk so a dead server does not take your history with it.

How do I search for a specific request across services?

Attach the trace or request ID as structured metadata at the agent, then filter on it. You get the high-cardinality lookup you wanted without creating a stream per request.

Self-host or use a managed service?

Self-hosting a monolithic Loki is genuinely low effort. Self-hosting a highly available multi-tenant Loki is a real ongoing job, and at that point Grafana Cloud or a hosted alternative deserves an honest cost comparison including your time. The break-even is further out than vendors suggest and closer than enthusiasts admit.

The one thing to remember

Labels select, filters search. Every serious problem with centralized logging with Loki traces back to someone treating a label as a search field. Loki will accept it without complaint, and the cost arrives weeks later as a query that will not finish.

So before adding any label, multiply out how many streams it creates. If the answer is more than a few thousand, it belongs in structured metadata or in the log line. And check your cardinality on a schedule, because this degrades gradually and you will not notice the day it crosses the line.

Need a logging setup that stays fast?

Most Loki installations I get asked to look at were fine on day one and got slower every week. Work I take on:

• Diagnosing a slow Loki: cardinality analysis, query shape review, and a concrete list of labels to move.
• Migrating Promtail configurations to Grafana Alloy, including Docker and Kubernetes log collection.
• Building a centralized logging stack from scratch: Loki, Alloy, Grafana, object storage, retention and dashboards.
• Restructuring labels and introducing structured metadata without losing queryability.
• Retention and storage cost work: per-stream policies, agent-side dropping, object storage tiering.
• Log-based alerting through the ruler, wired into an existing Prometheus and Alertmanager setup.

Send me the output of logcli series '{}' --analyze-labels and a query that feels slow, and I will tell you where the problem is.

The post One Label Too Many: Centralized Logging With Loki Without Wrecking It appeared first on John Nessime.

docker compose up -d

A web application, database, cache, monitoring service, and reverse proxy can all be launched in seconds. Everything is isolated, repeatable, and relatively easy to move between environments.

Then, eventually, something breaks.

A container refuses to start. A port is already occupied. A database loses access to its storage directory. One service cannot resolve the hostname of another. A reverse proxy returns a 502 error even though the application appears to be running.

That is usually the point where Docker stops feeling like a convenient deployment tool and starts becoming a real infrastructure skill.

This guide is based on the way I approach Docker issues on real Linux servers. It is not a list of commands to copy blindly. It is a troubleshooting method for discovering what failed, understanding why it failed, and solving it without making the situation worse.

Docker Is Usually Not the Real Problem

When a container fails, Docker is often blamed first. In many cases, however, Docker is only exposing a problem somewhere else in the system.

• Incorrect Linux file permissions
• A missing environment variable
• An unavailable database
• A port conflict on the host
• An incorrect volume mount
• A DNS or network configuration problem
• A reverse proxy pointing to the wrong destination
• Insufficient memory or disk space
• An application-level configuration error

The container runtime may be working exactly as designed. The application inside the container may simply be unable to operate under the conditions it was given.

This distinction matters because repeatedly deleting and rebuilding containers rarely fixes the underlying issue. It may temporarily hide it, but the same failure usually returns.

My First Rule of Docker Troubleshooting: Do Not Delete Anything Yet

When a container suddenly exits, the first instinct is often to remove it, rebuild the image, or restart the entire stack. That destroys useful evidence.

Before changing anything, I start by collecting information.

docker ps -a

This shows running and stopped containers, their current states, exposed ports, names, and recent exit information.

docker logs container-name

The logs frequently reveal the immediate cause: a missing file, invalid password, failed database connection, permission error, or malformed configuration.

docker inspect container-name

Inspection provides the container’s network settings, environment variables, volume mounts, restart policy, health status, image information, and runtime configuration.

These three commands answer a large percentage of Docker troubleshooting questions before any corrective action is required.

Understanding Docker Container States

A container can exist in several states, and each state tells a different story.

Running

The main process inside the container is active. This does not automatically mean the application is healthy or ready to receive traffic.

Exited

The container’s main process stopped. It may have completed normally, crashed, or failed during startup. The exit code and logs provide the next clues.

Restarting

The container repeatedly starts and fails while Docker applies its restart policy. This commonly indicates an application crash, invalid configuration, inaccessible dependency, or permission problem.

Created

The container was created but its main process has not started successfully.

Dead

Docker could not properly stop or remove the container. This is less common and may indicate a lower-level runtime, storage, or host issue.

Knowing the state prevents random troubleshooting. A networking issue requires a different investigation from a process that exits because of a syntax error.

Start With the Container Logs

Logs are normally the fastest route to the cause of a Docker issue.

docker logs --tail 100 container-name

This displays the latest 100 lines without flooding the terminal with the container’s entire history.

docker logs -f container-name

The follow option streams new log entries in real time. It is useful while restarting a service or reproducing a problem.

docker logs --since 30m container-name

This limits the output to a recent period, which is especially useful for long-running containers.

When reading logs, I do not look only at the final line. The true cause often appears several lines earlier. The last error may simply be a consequence of the first failure.

Docker Networking: Where Many Problems Begin

Docker networking is one of the most common sources of confusion, especially when several containers need to communicate.

Imagine a stack containing:

• Grafana
• Prometheus
• A web application
• MySQL or PostgreSQL
• Redis
• Nginx or Apache

Each service may work independently while communication between them fails.

The Localhost Mistake

Inside a container, localhost refers to that container itself. It does not refer to the Docker host, and it does not refer to another container.

For example, if Grafana needs to connect to Prometheus, the data source address should usually use the Docker Compose service name:

http://prometheus:9090

Using the following from inside the Grafana container would normally point back to Grafana itself:

http://localhost:9090

This small misunderstanding causes a surprising number of connection failures.

Confirm That Containers Share a Network

docker network ls

docker network inspect network-name

The inspection output shows which containers are attached to a network and which IP addresses were assigned.

Containers on unrelated networks cannot automatically communicate by service name. They must share a Docker network or use another explicitly configured route.

Test Connectivity From Inside the Container

docker exec -it container-name sh

Depending on the image, Bash may be available instead:

docker exec -it container-name bash

From inside the container, test name resolution and connectivity using the utilities available in that image:

getent hosts database
curl http://application:8080
wget -qO- http://prometheus:9090/-/ready

A successful result confirms that Docker DNS and network routing are functioning. A failed result narrows the investigation considerably.

Host Ports and Container Ports Are Different

A Docker Compose port mapping such as the following is read from left to right:

ports:
  - "8080:80"

Port 8080 belongs to the Docker host. Port 80 belongs to the application inside the container.

Traffic sent to the host on port 8080 is forwarded to port 80 inside the container.

If another service already occupies host port 8080, Docker cannot bind to it.

ss -tulpn

To check a specific port:

ss -tulpn | grep :8080

A common mistake is changing the container’s internal port when only the host-side mapping needs to change.

For example, this avoids a conflict while leaving the application untouched:

ports:
  - "8081:80"

Published Ports Are Not Required for Container-to-Container Traffic

Containers connected to the same Docker network can usually communicate through their internal ports without publishing those ports to the host.

For example, Prometheus can communicate with an exporter on the shared Docker network without exposing that exporter publicly.

This reduces unnecessary exposure and produces a cleaner architecture. Publish only the ports that must be reached from the host, a reverse proxy, or an external client.

Docker Volumes: Persistent Storage Done Properly

Containers should be treated as disposable. Persistent data should not depend on the writable filesystem inside a container.

Without a correctly configured volume, deleting a database container may also delete the database stored inside it.

Named Volumes

services:
  database:
    image: mariadb:latest
    volumes:
      - database-data:/var/lib/mysql

volumes:
  database-data:

Docker manages the storage location while the application writes to its expected internal directory.

Bind Mounts

volumes:
  - ./config:/etc/application

A bind mount maps a specific host path into the container. It is useful for configuration files, development directories, and data that administrators need to access directly.

Bind mounts also introduce more responsibility. The host directory must exist with the correct permissions, ownership, security context, and contents.

Inspect Existing Volumes

docker volume ls

docker volume inspect volume-name

Before deleting an apparently unused volume, confirm which application created it and whether it contains irreplaceable data.

Permission Problems Are Quiet but Destructive

Permission errors can prevent databases from initializing, web applications from uploading files, monitoring tools from writing data, or services from reading configuration files.

The first checks are simple:

ls -lah
stat path-to-file
stat path-to-directory

The user inside the container may have a different numeric user ID from the owner of the host directory. The names do not matter as much as the numeric UID and GID.

docker exec container-name id

Compare that result with the ownership of the mounted host path.

Changing permissions to 777 may appear to solve the issue, but it is usually an unsafe shortcut rather than a proper solution. Correct ownership and the minimum required permissions are preferable.

SELinux Can Affect Docker Bind Mounts

On distributions such as AlmaLinux, Rocky Linux, CentOS Stream, Fedora, and Red Hat Enterprise Linux, standard Unix permissions may look correct while SELinux still blocks access.

Docker Compose bind mounts may require an SELinux relabeling option:

volumes:
  - ./data:/var/lib/application:Z

The uppercase Z assigns a private SELinux label for the container. A lowercase z is used when the content must be shared between multiple containers.

SELinux should not be disabled simply to make a container work. The better approach is to identify the denied access and apply the correct labeling or policy.

Environment Variables Can Break an Entire Stack

Modern containerized applications depend heavily on environment variables for database credentials, API keys, application URLs, encryption secrets, feature flags, and runtime settings.

One missing or incorrectly escaped value can stop a service from starting.

docker compose config

This command renders the resolved Docker Compose configuration. It helps reveal missing substitutions, merged settings, malformed values, and unexpected defaults.

To inspect the environment available inside a running container:

docker exec container-name env

Be careful when sharing this output because it may include passwords, tokens, and private service credentials.

Docker Compose Configuration Errors

YAML is readable, but it is sensitive to indentation and structure. A service can be valid YAML while still being configured incorrectly for Docker Compose.

Before deploying a change, validate the file:

docker compose config --quiet

If the command returns without an error, the configuration is structurally valid.

For a complete rendered version:

docker compose config

This is especially valuable when Compose files use environment substitutions, overrides, extension fields, or multiple configuration files.

Health Checks Reveal More Than Running Status

A container can be running while the application inside it is unusable.

A database may still be initializing. A web service may be listening but unable to connect to its database. An API may be running but returning errors for every request.

A health check lets Docker test actual application readiness.

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 20s

The application should expose a lightweight endpoint that confirms its critical dependencies are available.

To inspect the current status:

docker inspect --format='{{json .State.Health}}' container-name

A health check does not repair an unhealthy service by itself, but it provides a far more accurate operational signal than process status alone.

Restart Policies Can Hide Repeated Failures

Restart policies are useful in production, but they can create the impression that a service is stable when it is actually crashing continuously.

restart: unless-stopped

If the application exits every few seconds, Docker restarts it every few seconds. The container may appear in the process list, but the service may never become ready.

Check the restart count:

docker inspect --format='{{.RestartCount}}' container-name

A rapidly increasing number indicates that the restart policy is masking an unresolved failure.

Reverse Proxy Errors: 502, 504, SSL Problems, and Redirect Loops

Reverse proxies such as Nginx, Apache, Traefik, and Caddy often sit between users and containerized applications.

When the public website fails, the application container may still be perfectly healthy. The proxy may simply be unable to reach it.

Common Reverse Proxy Problems

• The upstream hostname is incorrect.
• The proxy uses the host port when it should use the container port.
• The proxy and application do not share a Docker network.
• The application listens only on 127.0.0.1 inside the container.
• The proxy forwards HTTP while the backend expects HTTPS.
• The application is unaware that the original request used HTTPS.
• The public hostname does not match the certificate.
• Both the proxy and application force redirects, creating a loop.

Test the application directly before blaming the proxy:

curl -I http://127.0.0.1:published-port

If the application responds directly but fails through the domain, the reverse proxy, firewall, DNS, or TLS configuration becomes the main area of investigation.

An Application Must Listen on the Correct Interface

An application inside a container may start successfully while listening only on 127.0.0.1. In that situation, Docker’s network interface may not be able to reach it.

Containerized web applications should usually listen on:

0.0.0.0

This allows the process to accept connections through the container’s network interfaces.

This setting is commonly controlled by application arguments or environment variables such as HOST, BIND_ADDRESS, or LISTEN_ADDRESS.

Resource Exhaustion Can Look Like a Docker Failure

Containers share the CPU, memory, storage, and network resources of the host. One service can consume enough resources to affect every other workload.

docker stats

This provides a live view of CPU, memory, network, and block I/O usage for running containers.

On the host, I also check:

free -h
df -h
df -i
uptime

Disk space is not the only storage limit. A filesystem can also run out of inodes, preventing new files from being created even when gigabytes remain available.

Kernel logs may reveal out-of-memory termination:

dmesg | grep -i -E "out of memory|killed process|oom"

If the kernel killed the process, the application logs may end suddenly without a useful explanation.

Docker Disk Usage Grows Quietly

Old images, stopped containers, build cache, unused volumes, and unbounded logs can gradually consume the host’s storage.

docker system df

For more detail:

docker system df -v

Cleanup commands should be used carefully:

docker image prune
docker container prune
docker builder prune

A broad command such as docker system prune can remove more than expected. Never add the volume-removal option unless the contents of unused volumes have been verified and backed up.

Container Logs Can Fill the Server

Docker’s default JSON logging driver can create large log files when applications produce continuous output.

Log rotation can be configured in Docker Compose:

logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "5"

This limits the number and size of local log files for that container.

For production systems, centralized logging with tools such as Grafana Loki, Elasticsearch, OpenSearch, or a managed logging service makes incident investigation considerably easier.

Image Architecture and Compatibility Problems

An image built for one CPU architecture may not run correctly on another. This becomes relevant when moving between AMD64 servers, ARM-based cloud instances, Apple Silicon machines, and single-board computers.

Check the host architecture:

uname -m

Inspect the image:

docker image inspect image-name

Errors mentioning an invalid executable format frequently indicate an architecture mismatch.

Multi-platform builds can be produced using Docker Buildx:

docker buildx build --platform linux/amd64,linux/arm64 .

A New Image Tag Does Not Always Mean a Safe Upgrade

Using latest may be convenient, but it makes deployments less predictable. A future image update can introduce a breaking configuration change, database migration, removed feature, or incompatible dependency.

Production deployments are usually safer with an explicit version:

image: grafana/grafana:12.0.2

Before upgrading:

• Read the release notes.
• Confirm supported upgrade paths.
• Back up persistent data.
• Record the current working version.
• Test the change outside production where possible.
• Prepare a rollback procedure.

Containers make software easier to replace, but they do not make every application upgrade reversible.

Dependency Startup Order Is Not Readiness

Docker Compose can start services in a defined order, but starting a database container does not mean the database is immediately ready for connections.

An application may launch too early, fail to connect, and exit.

Health checks and dependency conditions can improve this:

services:
  database:
    image: postgres:17
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  application:
    image: example/application:1.0
    depends_on:
      database:
        condition: service_healthy

The application itself should still handle temporary dependency failures gracefully. Networks fluctuate, databases restart, and external services occasionally become unavailable.

Backups Matter More Than Recreating Containers

Application containers are normally easy to recreate. Persistent business data is not.

A reliable Docker backup strategy should cover:

• Database-native backups
• Named volumes and bind-mounted data
• Docker Compose files
• Environment files and secrets
• Reverse proxy configurations
• TLS and certificate configuration
• Application-specific settings
• A tested restoration procedure

A backup that has never been restored is only an assumption.

Database backups should normally use the database’s own tools rather than copying active database files blindly. Examples include pg_dump for PostgreSQL and mysqldump or mariadb-dump for MySQL-compatible systems.

Monitoring Changes Troubleshooting From Guesswork to Evidence

Without monitoring, administrators usually investigate after users report a failure. By then, the container may have restarted and the original conditions may no longer be visible.

Metrics help answer better questions:

• When did memory consumption begin increasing?
• Which container experienced the first error?
• Was the disk already nearly full?
• Did network latency rise before the application failed?
• How many times did the service restart?
• Did the host run out of memory?
• Was the reverse proxy still able to reach the backend?

Prometheus, Grafana, node-level exporters, container metrics, uptime probes, and centralized logs provide the historical context that individual Docker commands cannot.

A Practical Docker Troubleshooting Workflow

When a Docker service fails, I use a sequence rather than jumping between unrelated commands.

1. Identify the Exact Symptom

• Does the container fail to start?
• Does it start and then exit?
• Is it running but unreachable?
• Is the application responding with errors?
• Is the public domain failing while the local service works?
• Is the problem intermittent?

2. Check Container Status

docker ps -a

3. Read the Logs

docker logs --tail 200 container-name

4. Inspect Runtime Configuration

docker inspect container-name

5. Validate Docker Compose

docker compose config

6. Check Host Resources

docker stats
free -h
df -h
df -i

7. Test Internal Connectivity

Enter the relevant container and test DNS resolution, ports, and HTTP endpoints from the same network context as the application.

8. Check Volumes and Permissions

Confirm that mounts point to the expected locations and that the container user can read or write them as required.

9. Isolate the Reverse Proxy

Test the backend directly. If the backend works, investigate the proxy, DNS, TLS, firewall, and forwarded headers.

10. Change One Thing at a Time

Multiple simultaneous changes make it difficult to identify the actual solution. Apply one correction, test it, and record the result.

Commands I Frequently Use During Docker Investigations

# Show running containers
docker ps

# Show all containers
docker ps -a

# Read recent logs
docker logs --tail 100 container-name

# Follow logs
docker logs -f container-name

# Inspect a container
docker inspect container-name

# Enter a container
docker exec -it container-name sh

# View live resource use
docker stats

# List networks
docker network ls

# Inspect a network
docker network inspect network-name

# List volumes
docker volume ls

# Inspect a volume
docker volume inspect volume-name

# Validate Compose configuration
docker compose config

# View Compose service status
docker compose ps

# Restart one service
docker compose restart service-name

# Recreate one service
docker compose up -d --force-recreate service-name

# View Docker disk usage
docker system df

# View host listening ports
ss -tulpn

# Check disk usage
df -h

# Check inode usage
df -i

# Check memory
free -h

Docker Troubleshooting Mistakes to Avoid

Deleting Containers Before Reading Their Logs

This removes evidence and often leaves the original problem unresolved.

Using Docker System Prune Without Reviewing the Impact

Cleanup commands can remove useful images, build cache, stopped containers, and potentially important storage.

Using Latest Everywhere

Uncontrolled image updates reduce reproducibility and can introduce breaking changes unexpectedly.

Opening Every Port Publicly

Internal services should remain internal unless external access is genuinely required.

Setting Permissions to 777

This weakens security and avoids fixing the actual ownership or access-control issue.

Restarting the Entire Server Too Early

A restart may temporarily restore service while destroying evidence about the original failure.

Changing Several Variables at Once

Even when the issue disappears, it becomes impossible to know which change solved it.

Docker Compose Is Also Infrastructure Documentation

A well-organized Compose file documents the structure of an application environment.

• Which services exist
• Which image versions are deployed
• Which networks connect them
• Which ports are externally accessible
• Where persistent data is stored
• Which environment variables are required
• Which services depend on others
• Which health checks are available
• How containers restart after failure

Six months after a deployment, this information is often more useful than a separate document that was never updated.

Readable service names, comments for non-obvious decisions, pinned image versions, organized environment files, and clear volume names make future troubleshooting much faster.

What Docker Failures Actually Teach

Some of the most useful infrastructure lessons come from deployments that did not work as expected.

A failed container may teach you how Linux permissions interact with numeric user IDs. A 502 error may reveal the difference between host ports and container ports. A disappearing database may demonstrate why persistent volumes matter. An unreachable service may expose a misunderstanding about Docker DNS or network boundaries.

These incidents are frustrating, but they build practical knowledge that clean tutorial environments rarely provide.

Production systems do not normally fail in textbook ways. Several small issues may combine: a nearly full disk, oversized logs, an automatic image upgrade, and an aggressive restart policy can create one confusing incident.

Final Thoughts

Docker makes application deployment more portable, repeatable, and manageable, but it does not remove the need to understand the systems underneath it.

Effective Docker troubleshooting requires knowledge of Linux, networking, storage, process management, application configuration, security, and monitoring.

The most important habit is simple: follow the evidence.

Check the container state. Read the logs. Inspect the configuration. Test the network from the correct location. Verify volumes and permissions. Check the host’s resources. Separate the application from the proxy. Change one thing at a time.

The next time a Docker container fails, resist the urge to delete everything and begin again. The answer is usually already present in the logs, runtime configuration, network design, storage mapping, or host environment.

Docker troubleshooting becomes much less intimidating once every failure is treated as an investigation rather than an emergency.

Frequently Asked Questions About Docker Issues

Why does my Docker container exit immediately?

A Docker container exits when its main process stops. Common causes include invalid configuration, missing environment variables, permission errors, failed dependency connections, and incorrect startup commands. Use docker logs and docker inspect to identify the cause.

Why is my Docker container running but not accessible?

The application may be listening on the wrong interface, the port may not be published correctly, a firewall may be blocking traffic, or the reverse proxy may point to the wrong hostname or port. Test the application both inside the container and directly through the host port.

Why can one Docker container not connect to another?

The containers may not share a Docker network, the wrong service name may be used, or the destination service may not be listening on the expected port. Inside Docker, use the Compose service name rather than localhost.

How do I find which process is using a Docker port?

Run ss -tulpn on the Docker host and filter for the relevant port. If the host port is already occupied, change the host side of the Docker port mapping or stop the conflicting service.

Can restarting Docker solve container problems?

Restarting Docker may temporarily restore a service, but it can also hide the original cause. Logs, container state, disk space, memory, network connectivity, and volume permissions should be checked before restarting the Docker daemon or server.

How can I prevent Docker logs from filling the disk?

Configure log rotation using the Docker logging options, such as max-size and max-file. Production environments may also benefit from centralized logging with Grafana Loki, OpenSearch, Elasticsearch, or a managed logging platform.

Should I use the latest Docker image tag?

The latest tag is convenient but unpredictable. Pinning a tested image version improves repeatability and makes upgrades and rollbacks easier to control.

Are Docker volumes automatically backed up?

No. Docker volumes provide persistent storage, but Docker does not automatically create external backups. Important volumes and databases need a separate, tested backup and restoration process.

The post Docker Issues & Troubleshooting: Real-World Solutions appeared first on John Nessime.