Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker’s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.

This post covers Docker log rotation properly: what the defaults actually do, why the daemon.json change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.

The default is unbounded, and that is documented behaviour

Docker’s default logging driver is json-file. It captures stdout and stderr from the container’s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.

The important part is the option defaults. For json-file, max-size defaults to unlimited, max-file defaults to 1, and compress defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.

Docker’s own documentation is upfront about why. Keeping json-file unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.

Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:

# What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"

The gap between the first command and the second is where most of the pain lives, and we will come back to it.

To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container’s log file plus a small amount of metadata, so the sizes are close enough to be useful:

sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10

Container IDs are not memorable, so map the winner back to a name with docker ps --no-trunc or by grepping the ID against docker ps -aq. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.

Setting up Docker log rotation on the daemon

The daemon-wide fix goes in /etc/docker/daemon.json. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

What each key buys you:

• max-size caps a single file. Accepts an integer plus k, m or g. This is the setting that stops the runaway.
• max-file caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is only effective if max-size is also set, which is the single most common misconfiguration I see.
• compress gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.

Two things trip people up here. First, every value in log-opts must be a JSON string, including the numeric ones. Writing "max-file": 3 without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed daemon.json on a production host is a much worse afternoon than a full disk:

# Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  && echo "daemon.json parses OK"

sudo systemctl restart docker

Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is max-size multiplied by max-file. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.

Why your daemon.json change did nothing

This is the failure mode that costs the most time, because it looks exactly like the fix not working.

Logging configuration is baked into a container’s host config when the container is created. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because docker restart stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.

So the container that filled your disk keeps filling your disk, its log file still has no cap, and docker info cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.

To actually apply it, recreate the container and then verify rather than assuming:

# Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web

You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.

Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.

Pinning limits per service in Compose

Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides daemon.json entirely. A YAML anchor keeps it from turning into copy-paste sprawl:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

Be aware that the logging block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with its defaults, which for json-file means straight back to unbounded. Half-specifying is worse than not specifying at all.

You deleted the log file and df did not move

Classic 3am mistake, and it is worth understanding rather than memorising.

When you rm the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from ls, the daemon keeps writing into a file you can no longer see, and df reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.

truncate -s 0 is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:

sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"

There is a cost. Truncating out from under the daemon can leave an in-flight docker logs -f stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.

The same reasoning applies to logrotate rules pointed at /var/lib/docker/containers/*/*.log. Rotating with copytruncate works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker’s own docs warn against external tools touching those files. If the driver can rotate for you, let it.

Choosing a driver, and what each one costs

json-file

The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. docker logs works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.

local

Docker’s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it rotates and compresses by default, retaining five files of 20 MB each per container out of the box. docker logs works exactly as normal. The trade-off is that the format is Docker’s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is docker logs plus a shipper that talks to the daemon, this is the safest default you can pick.

journald

Hands container output to systemd’s journal, which means you get journalctl filtering, structured fields, and the journal’s own size management via SystemMaxUse in journald.conf. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.

The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at RateLimitIntervalSec and RateLimitBurst before you need them, not after.

syslog, fluentd and the remote drivers

These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.

What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.

none

Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because docker logs returns nothing and you will waste twenty minutes concluding the container is broken.

The blocking behaviour worth knowing about

By default the logging path is blocking. If the driver cannot keep up, the container’s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.

docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1

Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.

Dual logging, and why docker logs still works

If you switch to a remote driver and find docker logs still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the local driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.

Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.

Reclaiming space when the disk is already full

Order matters here. Free space first, fix the cause second, verify third.

1. Confirm logs are actually the problem. Compare du -sh /var/lib/docker/containers against docker system df, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.
2. Identify the largest log files with the du and sort command from earlier.
3. Copy anything you need out first, if the logs are part of an active investigation.
4. Truncate the offenders with truncate -s 0. Never rm.
5. Set the limits in daemon.json, validate the JSON, restart the daemon.
6. Recreate the containers so the limits take effect, then confirm with docker inspect on each one.
7. Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.

Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.

Troubleshooting

Rotation is configured but files still grow past the cap. The container predates the change. Check docker inspect --format '{{.HostConfig.LogConfig}}' and recreate it.

The daemon will not start after editing daemon.json. Almost always invalid JSON, or unquoted values in log-opts. Read the actual error with journalctl -u docker --no-pager -n 50 rather than guessing.

max-file is set but only one file ever appears. max-file does nothing without max-size. Set both.

docker logs -f hangs after showing existing output. Something truncated the file underneath the daemon, usually an external logrotate rule. Reattach to recover, then remove the rule.

docker logs returns nothing at all. Check the driver. none discards everything, and some remote drivers only serve reads through the dual-logging cache.

Disk usage did not drop after deleting log files. The daemon still holds the descriptors. Restart the daemon to release the inodes, and use truncate next time.

Logs vanish sooner than the retention you configured. On journald, check journal rate limiting and SystemMaxUse. On a remote driver in non-blocking mode, check whether the buffer is overflowing.

Common mistakes

• Assuming a daemon restart applies the new limits to running containers. It does not.
• Setting max-file without max-size and believing rotation is on.
• Writing numeric values unquoted in log-opts.
• Overwriting an existing daemon.json instead of merging, and silently dropping storage or network settings.
• Using rm on a live container’s log file and concluding the disk report is broken.
• Specifying a logging.driver in Compose without its options, which resets that service back to the driver’s own unbounded defaults.
• Pointing host logrotate at the Docker containers directory when the driver could have handled it.
• Sizing rotation per container without multiplying by container count against the actual disk.

Best practices

• Set logging limits in daemon.json as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.
• Prefer the local driver on standalone hosts unless something specifically needs to read the JSON files.
• Turn on compress. Log text compresses well enough that you get more retention for the same disk.
• Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.
• Do the multiplication. Worst-case host usage is max-size × max-file × container count, and it should be a fraction of your disk, not most of it.
• Alert on Docker data root usage at a threshold that leaves you time to act.
• Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.
• Verify with docker inspect after any change. docker info tells you the daemon’s intent, not what your containers carry.

Frequently asked questions

Does Docker rotate container logs automatically?

Not with the default driver. json-file ships with max-size unlimited and max-file set to 1, so a single file grows without bound. The local driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.

Where are Docker container logs stored on the host?

Under the Docker data root, one directory per container, which on a standard Linux install means /var/lib/docker/containers/. Rather than assembling the path by hand, ask the daemon: docker inspect --format '{{.LogPath}}' <container>. That works regardless of a custom data-root.

How do I clear Docker logs without stopping the container?

Truncate the file to zero bytes with truncate -s 0 on the path from {{.LogPath}}. Do not use rm: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.

Why did my daemon.json rotation settings not apply?

Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and docker restart reuses the same container object. Recreate the container, then confirm with docker inspect --format '{{.HostConfig.LogConfig}}'.

Should I use the local driver or json-file?

Use local if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use json-file if an agent or shipper on the host tails those files, because it cannot parse the local format. Either way docker logs behaves the same.

Can I use logrotate for Docker container logs instead?

You can, and plenty of hosts do, but the driver options are the better tool. A copytruncate rule pointed at the containers directory reaches into files the daemon treats as private, and can leave docker logs -f streams stalled after each rotation. Use logrotate for logs your applications write to mounted volumes, and let the driver handle stdout.

How much disk should I budget for container logs?

Worst case is max-size × max-file per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.

The one thing worth remembering

Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.

So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with docker inspect rather than trusting docker info. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.

Need a hand with container logging on your hosts?

Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:

• Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits
• Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number
• Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back
• Choosing between json-file, local, journald and shipping off-box, including the blocking behaviour trade-offs under load
• Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends
• Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot

If you want a second opinion, send me your daemon.json, the output of docker ps and a du -sh of your containers directory, and I will tell you what I would change.

The post Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation appeared first on John Nessime.

Nothing about the Loki config changed. What changed is that three weeks ago somebody added request_id as a label because it seemed helpful, and Loki has been quietly building millions of separate streams ever since. The index that used to fit comfortably in memory does not. Every query now has to open thousands of tiny chunks instead of a handful of large ones.

This is the failure mode that defines centralized logging with Loki, and it is almost entirely self-inflicted. Loki is cheap and fast when you use it the way it was designed and miserable when you bring habits from Elasticsearch. Nothing warns you at the moment you make the mistake. The bill comes weeks later, during an incident.

This covers the model you have to internalise, how to choose labels, what replaced Promtail (it is gone, and every tutorial older than this year is wrong about it), the LogQL patterns worth knowing, and how to keep retention and storage costs under control.

Loki is not Elasticsearch, and that is the entire design

Elasticsearch builds a full inverted index over every token in every log line. Powerful, and the index often ends up as large as the logs themselves. That is where the cost comes from.

Loki indexes only labels. The log content is compressed into chunks and pushed to object storage, untouched and unindexed. A query does two things: it uses the index to select which streams to read, then it brute-force scans the contents of those chunks in parallel.

The consequence is a rule worth writing on a wall: labels select, filters search. Labels exist to narrow down which chunks get opened. Everything else is a filter applied at query time. Scanning is genuinely fast because it is embarrassingly parallel, so filtering on content is not the expensive operation people assume.

What is expensive is streams. A stream is one unique combination of label values, and each gets its own chunks. Do the arithmetic before you add a label:

• 8 jobs, 3 environments, 12 hosts gives you 288 streams. Comfortable.
• Add request_id, and you multiply that by however many requests you serve. The index explodes, chunks get flushed while nearly empty, and object storage fills with millions of tiny files that queries must open individually.

Ingesters also hold every active stream in memory. Cardinality does not just make queries slow; it is how Loki gets OOM-killed.

Choosing labels

A label earns its place if it satisfies all three: the set of values is bounded and small, you know the values in advance, and you would actually use it to narrow a search.

• Good: job, service, env, host, namespace, level.
• Bad: request_id, trace_id, user_id, session_id, full URL paths, IP addresses, timestamps, durations, anything derived from the log content itself.

Two subtler traps. instance including an ephemeral port is unbounded. And in container environments, pod names churn constantly, so labelling by pod name gives you a new stream every deployment forever.

Aim to keep the total under a few thousand active streams. Check rather than guess, using logcli, which reports cardinality per label:

# Which of your labels is quietly out of control?
logcli series '{}' --analyze-labels

Run that before you have a problem. It is the single most useful diagnostic here and almost nobody knows about it.

The escape hatch: structured metadata

“Never label a trace ID” is correct but unsatisfying, because sometimes you genuinely need to find every line for one trace. Loki 3 answers this with structured metadata: key-value pairs attached to a log line, stored alongside it, not in the index and not creating streams.

limits_config:
  # Off by default on older configs. Turn it on before your
  # agent starts sending structured metadata, or the writes
  # get rejected and you spend an hour puzzling over it.
  allow_structured_metadata: true

Query it with the same label-filter syntax, and it costs you nothing in cardinality:

{job="api", env="production"} | trace_id="9f2c1b4e"

So the decision becomes three-way rather than two-way: bounded and used for selection becomes a label, high-cardinality but frequently searched becomes structured metadata, and everything else stays in the line and gets parsed at query time.

Promtail is gone. Use Alloy.

This one has a date and it matters, because it invalidates most of the Loki material on the internet.

Promtail, the agent every Loki tutorial tells you to install, was deprecated in February 2025 and reached end of life on 2 March 2026. No security patches, no bug fixes, no updates. Grafana Alloy, their distribution of the OpenTelemetry Collector, is the replacement, and it handles logs, metrics, traces and profiles in one binary instead of running a separate agent per signal.

If you already run Promtail, there is a conversion tool rather than a rewrite:

# Converts an existing Promtail config to Alloy syntax.
# Read the output before deploying it; the converter flags
# anything it could not translate cleanly.
alloy convert --source-format=promtail --output=config.alloy promtail.yaml

Alloy configuration is a component graph rather than a YAML pipeline. Components reference each other’s outputs, which reads oddly at first and then makes more sense than nested YAML stages:

// Find the files, and attach the labels that will select them later.
// Note what is NOT here: nothing derived from log content.
local.file_match "system" {
  path_targets = [{
    __path__ = "/var/log/syslog",
    job      = "syslog",
    host     = constants.hostname,
    env      = "production",
  }]
}

loki.source.file "system" {
  targets    = local.file_match.system.targets
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Between source and write you can insert a loki.process component to parse lines, drop noise, and promote fields into structured metadata. Do the dropping here rather than at the Loki end: logs you never ship cost nothing to store, index or query, and debug-level chatter from a busy service is usually the single largest slice of ingest.

For Docker hosts, Alloy has components that discover containers and read their logs directly, which is tidier than tailing the JSON files under /var/lib/docker and surviving a logging-driver change.

Deployment shape and storage

Loki ships as one binary that runs in different modes depending on the -target flag.

• Monolithic. Everything in one process. Right for a single VPS, a homelab, or anything up to a modest daily volume. Do not feel bad about this; a lot of people run microservices mode for a workload one process would handle.
• Simple scalable. Splits into read, write and backend targets behind a load balancer. The sensible next step, and where most self-hosted setups should stop.
• Microservices. Every component separately. Genuinely necessary at large scale and a substantial operational burden below it.

For storage, use TSDB with schema v13. Older guides show boltdb-shipper; that is the previous generation. Chunks belong in object storage in anything you care about, because filesystem storage means one disk, one node, and no redundancy.

schema_config:
  configs:
    # Schema changes are additive. Add a new entry with a future
    # 'from' date rather than editing the existing one, or Loki
    # cannot read the data it already wrote.
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

That “additive” note is worth taking seriously. Editing an existing schema entry in place is one of the few ways to make historical logs unreadable.

Any S3-compatible store works, which is where the cost advantage lives. Chunks are compressed and write-once, so cheaper tiers suit them well: S3 itself, or MinIO if you want it self-hosted, or one of the cheaper compatible providers like Backblaze B2 or Cloudflare R2. Watch egress pricing rather than storage pricing, since queries read chunks back.

LogQL worth actually knowing

Every query starts with a stream selector in braces. That part is mandatory and it is what decides how much data gets read.

# Order matters for performance:
#   1. select streams (index lookup)
#   2. filter lines (cheap string match, runs on every line)
#   3. parse (expensive, so run it on the smallest set possible)
{job="api", env="production"} |= "timeout" | json | status="500"

People routinely write | json before the line filter and then wonder why the query is slow. Parsing every line in the range to discard most of them is exactly the wrong order.

Loki also turns logs into metrics, which is where it earns its place next to Prometheus:

# Error rate per service, graphable and alertable.
sum by (job) (rate({env="production"} |= "level=error" [5m]))

# Which streams are eating your ingest budget?
topk(10, sum by (job, host) (count_over_time({env="production"}[1h])))

That second one is worth saving as a dashboard panel. Ingest problems announce themselves there long before they show up as a slow query.

Loki’s ruler evaluates the same expressions as alerting rules and sends them to Alertmanager, so “alert when this service logs more than N errors in five minutes” needs no extra component if you already run Prometheus.

Retention, and why it silently does nothing

Setting retention_period is not enough. Retention is enforced by the compactor, and the compactor does not do it unless you tell it to. Plenty of people set a retention period, watch their bucket keep growing, and never connect the two.

compactor:
  working_directory: /loki/compactor
  # Without this, the compactor compacts the index and
  # deletes nothing. This is the line people miss.
  retention_enabled: true
  delete_request_store: s3

limits_config:
  retention_period: 720h        # 30 days, global default

Per-stream retention lets you keep what matters and discard what does not: audit logs for a year, application debug logs for three days. That difference tends to matter more to the bill than any storage tier choice.

Deletion is asynchronous. Chunks disappear from object storage some time after the index entries go, so do not panic if the bucket does not shrink immediately.

Troubleshooting

Queries are slow or time out

Cardinality first. Run logcli series '{}' --analyze-labels and look for a label with a suspiciously large value count. Then check query shape: a selector that matches everything, a parser before a line filter, or a time range far wider than needed. Adding query frontend caching helps repeated queries but will not rescue a bad label set.

Stream limit exceeded

Loki is telling you the truth. Raising max_global_streams_per_user buys time and makes the underlying problem worse. Find the label that is generating streams and move it to structured metadata or into the line.

Rate limit or ingestion errors

Two different limits produce similar-looking rejections: a global ingestion rate, and a per-stream rate. Per-stream limits usually mean one very chatty stream rather than too much traffic overall, and the fix is to split it by a label or drop the noise at the agent.

Logs are not appearing at all

Work forward from the agent. Check the agent’s own logs first, then whether it can reach the push endpoint, then whether the file is actually readable by the agent’s user. Container log permissions and SELinux catch people constantly. In Grafana, remember the label browser only shows labels seen recently, so an empty dropdown often means nothing has arrived, not that the query is wrong.

Logs stop after an agent restart

The position file that tracks how far it read is not persisting. In a container, that means it is not on a mounted volume, so every restart re-reads or skips depending on configuration.

Storage keeps growing despite retention

Confirm retention_enabled: true is actually set, that exactly one compactor is running, and that it has write access to the object store. A compactor that cannot delete usually says so in its logs and nowhere else.

Common mistakes

• Treating labels like Elasticsearch fields and labelling everything.
• Promoting a parsed value into a label without checking how many distinct values it has.
• Following a tutorial that installs Promtail, which is now end of life.
• Writing | json before the line filter.
• Setting retention_period without retention_enabled: true.
• Running more than one compactor.
• Editing an existing schema_config entry instead of adding a new one.
• Filesystem storage on a single node for anything you would miss.
• Shipping every debug line and then paying to store, index and scan it.
• Raising stream limits instead of fixing cardinality.
• Running microservices mode for a workload the single binary handles comfortably.
• No dashboard showing ingest volume per stream, so growth is invisible until queries break.

Best practices

• Decide labels deliberately: bounded, known in advance, used for selection.
• Put high-cardinality but searchable values in structured metadata, not labels.
• Run --analyze-labels periodically, not just when something breaks.
• Drop noise at the agent, before it costs you anything downstream.
• Emit structured logs from your applications so parsers are cheap and reliable.
• Start monolithic and move to simple scalable only when you have a reason.
• TSDB, schema v13, chunks in object storage.
• Turn retention on properly and use per-stream retention to keep costs sane.
• Order queries: select, filter, parse.
• Alert on log-derived metrics through the ruler rather than eyeballing dashboards.
• Graph ingest volume per job so cardinality growth shows up as a trend.

FAQ

Is Loki a drop-in replacement for the ELK stack?

No, and expecting it to be is where most disappointment comes from. Loki is dramatically cheaper to run and much better suited to “show me the logs for this service around this time”. Elasticsearch is better at ad-hoc full-text search across everything with no idea where to start. Pick based on which of those you do more often.

Why is my Loki so slow?

Almost always label cardinality. Run logcli series '{}' --analyze-labels and look for the outlier. The second most common cause is a query that parses before it filters. Hardware is a distant third.

Should I still use Promtail?

No. It reached end of life in March 2026 and receives no security patches. Use Grafana Alloy, and run alloy convert against your existing config rather than rewriting it from scratch.

How many labels is too many?

The count matters less than the product of their distinct values. Five labels with a handful of values each is fine. Three labels where one has fifty thousand values is not. Multiply before you add.

Can I run Loki on a single VPS?

Yes, and for a handful of services it is a perfectly good answer. Monolithic mode alongside Grafana and Prometheus fits on a modest box from any provider. Point chunk storage at an object store rather than local disk so a dead server does not take your history with it.

How do I search for a specific request across services?

Attach the trace or request ID as structured metadata at the agent, then filter on it. You get the high-cardinality lookup you wanted without creating a stream per request.

Self-host or use a managed service?

Self-hosting a monolithic Loki is genuinely low effort. Self-hosting a highly available multi-tenant Loki is a real ongoing job, and at that point Grafana Cloud or a hosted alternative deserves an honest cost comparison including your time. The break-even is further out than vendors suggest and closer than enthusiasts admit.

The one thing to remember

Labels select, filters search. Every serious problem with centralized logging with Loki traces back to someone treating a label as a search field. Loki will accept it without complaint, and the cost arrives weeks later as a query that will not finish.

So before adding any label, multiply out how many streams it creates. If the answer is more than a few thousand, it belongs in structured metadata or in the log line. And check your cardinality on a schedule, because this degrades gradually and you will not notice the day it crosses the line.

Need a logging setup that stays fast?

Most Loki installations I get asked to look at were fine on day one and got slower every week. Work I take on:

• Diagnosing a slow Loki: cardinality analysis, query shape review, and a concrete list of labels to move.
• Migrating Promtail configurations to Grafana Alloy, including Docker and Kubernetes log collection.
• Building a centralized logging stack from scratch: Loki, Alloy, Grafana, object storage, retention and dashboards.
• Restructuring labels and introducing structured metadata without losing queryability.
• Retention and storage cost work: per-stream policies, agent-side dropping, object storage tiering.
• Log-based alerting through the ruler, wired into an existing Prometheus and Alertmanager setup.

Send me the output of logcli series '{}' --analyze-labels and a query that feels slow, and I will tell you where the problem is.

The post One Label Too Many: Centralized Logging With Loki Without Wrecking It appeared first on John Nessime.