Nothing about the Loki config changed. What changed is that three weeks ago somebody added request_id as a label because it seemed helpful, and Loki has been quietly building millions of separate streams ever since. The index that used to fit comfortably in memory does not. Every query now has to open thousands of tiny chunks instead of a handful of large ones.

This is the failure mode that defines centralized logging with Loki, and it is almost entirely self-inflicted. Loki is cheap and fast when you use it the way it was designed and miserable when you bring habits from Elasticsearch. Nothing warns you at the moment you make the mistake. The bill comes weeks later, during an incident.

This covers the model you have to internalise, how to choose labels, what replaced Promtail (it is gone, and every tutorial older than this year is wrong about it), the LogQL patterns worth knowing, and how to keep retention and storage costs under control.

Loki is not Elasticsearch, and that is the entire design

Elasticsearch builds a full inverted index over every token in every log line. Powerful, and the index often ends up as large as the logs themselves. That is where the cost comes from.

Loki indexes only labels. The log content is compressed into chunks and pushed to object storage, untouched and unindexed. A query does two things: it uses the index to select which streams to read, then it brute-force scans the contents of those chunks in parallel.

The consequence is a rule worth writing on a wall: labels select, filters search. Labels exist to narrow down which chunks get opened. Everything else is a filter applied at query time. Scanning is genuinely fast because it is embarrassingly parallel, so filtering on content is not the expensive operation people assume.

What is expensive is streams. A stream is one unique combination of label values, and each gets its own chunks. Do the arithmetic before you add a label:

• 8 jobs, 3 environments, 12 hosts gives you 288 streams. Comfortable.
• Add request_id, and you multiply that by however many requests you serve. The index explodes, chunks get flushed while nearly empty, and object storage fills with millions of tiny files that queries must open individually.

Ingesters also hold every active stream in memory. Cardinality does not just make queries slow; it is how Loki gets OOM-killed.

Choosing labels

A label earns its place if it satisfies all three: the set of values is bounded and small, you know the values in advance, and you would actually use it to narrow a search.

• Good: job, service, env, host, namespace, level.
• Bad: request_id, trace_id, user_id, session_id, full URL paths, IP addresses, timestamps, durations, anything derived from the log content itself.

Two subtler traps. instance including an ephemeral port is unbounded. And in container environments, pod names churn constantly, so labelling by pod name gives you a new stream every deployment forever.

Aim to keep the total under a few thousand active streams. Check rather than guess, using logcli, which reports cardinality per label:

# Which of your labels is quietly out of control?
logcli series '{}' --analyze-labels

Run that before you have a problem. It is the single most useful diagnostic here and almost nobody knows about it.

The escape hatch: structured metadata

“Never label a trace ID” is correct but unsatisfying, because sometimes you genuinely need to find every line for one trace. Loki 3 answers this with structured metadata: key-value pairs attached to a log line, stored alongside it, not in the index and not creating streams.

limits_config:
  # Off by default on older configs. Turn it on before your
  # agent starts sending structured metadata, or the writes
  # get rejected and you spend an hour puzzling over it.
  allow_structured_metadata: true

Query it with the same label-filter syntax, and it costs you nothing in cardinality:

{job="api", env="production"} | trace_id="9f2c1b4e"

So the decision becomes three-way rather than two-way: bounded and used for selection becomes a label, high-cardinality but frequently searched becomes structured metadata, and everything else stays in the line and gets parsed at query time.

Promtail is gone. Use Alloy.

This one has a date and it matters, because it invalidates most of the Loki material on the internet.

Promtail, the agent every Loki tutorial tells you to install, was deprecated in February 2025 and reached end of life on 2 March 2026. No security patches, no bug fixes, no updates. Grafana Alloy, their distribution of the OpenTelemetry Collector, is the replacement, and it handles logs, metrics, traces and profiles in one binary instead of running a separate agent per signal.

If you already run Promtail, there is a conversion tool rather than a rewrite:

# Converts an existing Promtail config to Alloy syntax.
# Read the output before deploying it; the converter flags
# anything it could not translate cleanly.
alloy convert --source-format=promtail --output=config.alloy promtail.yaml

Alloy configuration is a component graph rather than a YAML pipeline. Components reference each other’s outputs, which reads oddly at first and then makes more sense than nested YAML stages:

// Find the files, and attach the labels that will select them later.
// Note what is NOT here: nothing derived from log content.
local.file_match "system" {
  path_targets = [{
    __path__ = "/var/log/syslog",
    job      = "syslog",
    host     = constants.hostname,
    env      = "production",
  }]
}

loki.source.file "system" {
  targets    = local.file_match.system.targets
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Between source and write you can insert a loki.process component to parse lines, drop noise, and promote fields into structured metadata. Do the dropping here rather than at the Loki end: logs you never ship cost nothing to store, index or query, and debug-level chatter from a busy service is usually the single largest slice of ingest.

For Docker hosts, Alloy has components that discover containers and read their logs directly, which is tidier than tailing the JSON files under /var/lib/docker and surviving a logging-driver change.

Deployment shape and storage

Loki ships as one binary that runs in different modes depending on the -target flag.

• Monolithic. Everything in one process. Right for a single VPS, a homelab, or anything up to a modest daily volume. Do not feel bad about this; a lot of people run microservices mode for a workload one process would handle.
• Simple scalable. Splits into read, write and backend targets behind a load balancer. The sensible next step, and where most self-hosted setups should stop.
• Microservices. Every component separately. Genuinely necessary at large scale and a substantial operational burden below it.

For storage, use TSDB with schema v13. Older guides show boltdb-shipper; that is the previous generation. Chunks belong in object storage in anything you care about, because filesystem storage means one disk, one node, and no redundancy.

schema_config:
  configs:
    # Schema changes are additive. Add a new entry with a future
    # 'from' date rather than editing the existing one, or Loki
    # cannot read the data it already wrote.
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

That “additive” note is worth taking seriously. Editing an existing schema entry in place is one of the few ways to make historical logs unreadable.

Any S3-compatible store works, which is where the cost advantage lives. Chunks are compressed and write-once, so cheaper tiers suit them well: S3 itself, or MinIO if you want it self-hosted, or one of the cheaper compatible providers like Backblaze B2 or Cloudflare R2. Watch egress pricing rather than storage pricing, since queries read chunks back.

LogQL worth actually knowing

Every query starts with a stream selector in braces. That part is mandatory and it is what decides how much data gets read.

# Order matters for performance:
#   1. select streams (index lookup)
#   2. filter lines (cheap string match, runs on every line)
#   3. parse (expensive, so run it on the smallest set possible)
{job="api", env="production"} |= "timeout" | json | status="500"

People routinely write | json before the line filter and then wonder why the query is slow. Parsing every line in the range to discard most of them is exactly the wrong order.

Loki also turns logs into metrics, which is where it earns its place next to Prometheus:

# Error rate per service, graphable and alertable.
sum by (job) (rate({env="production"} |= "level=error" [5m]))

# Which streams are eating your ingest budget?
topk(10, sum by (job, host) (count_over_time({env="production"}[1h])))

That second one is worth saving as a dashboard panel. Ingest problems announce themselves there long before they show up as a slow query.

Loki’s ruler evaluates the same expressions as alerting rules and sends them to Alertmanager, so “alert when this service logs more than N errors in five minutes” needs no extra component if you already run Prometheus.

Retention, and why it silently does nothing

Setting retention_period is not enough. Retention is enforced by the compactor, and the compactor does not do it unless you tell it to. Plenty of people set a retention period, watch their bucket keep growing, and never connect the two.

compactor:
  working_directory: /loki/compactor
  # Without this, the compactor compacts the index and
  # deletes nothing. This is the line people miss.
  retention_enabled: true
  delete_request_store: s3

limits_config:
  retention_period: 720h        # 30 days, global default

Per-stream retention lets you keep what matters and discard what does not: audit logs for a year, application debug logs for three days. That difference tends to matter more to the bill than any storage tier choice.

Deletion is asynchronous. Chunks disappear from object storage some time after the index entries go, so do not panic if the bucket does not shrink immediately.

Troubleshooting

Queries are slow or time out

Cardinality first. Run logcli series '{}' --analyze-labels and look for a label with a suspiciously large value count. Then check query shape: a selector that matches everything, a parser before a line filter, or a time range far wider than needed. Adding query frontend caching helps repeated queries but will not rescue a bad label set.

Stream limit exceeded

Loki is telling you the truth. Raising max_global_streams_per_user buys time and makes the underlying problem worse. Find the label that is generating streams and move it to structured metadata or into the line.

Rate limit or ingestion errors

Two different limits produce similar-looking rejections: a global ingestion rate, and a per-stream rate. Per-stream limits usually mean one very chatty stream rather than too much traffic overall, and the fix is to split it by a label or drop the noise at the agent.

Logs are not appearing at all

Work forward from the agent. Check the agent’s own logs first, then whether it can reach the push endpoint, then whether the file is actually readable by the agent’s user. Container log permissions and SELinux catch people constantly. In Grafana, remember the label browser only shows labels seen recently, so an empty dropdown often means nothing has arrived, not that the query is wrong.

Logs stop after an agent restart

The position file that tracks how far it read is not persisting. In a container, that means it is not on a mounted volume, so every restart re-reads or skips depending on configuration.

Storage keeps growing despite retention

Confirm retention_enabled: true is actually set, that exactly one compactor is running, and that it has write access to the object store. A compactor that cannot delete usually says so in its logs and nowhere else.

Common mistakes

• Treating labels like Elasticsearch fields and labelling everything.
• Promoting a parsed value into a label without checking how many distinct values it has.
• Following a tutorial that installs Promtail, which is now end of life.
• Writing | json before the line filter.
• Setting retention_period without retention_enabled: true.
• Running more than one compactor.
• Editing an existing schema_config entry instead of adding a new one.
• Filesystem storage on a single node for anything you would miss.
• Shipping every debug line and then paying to store, index and scan it.
• Raising stream limits instead of fixing cardinality.
• Running microservices mode for a workload the single binary handles comfortably.
• No dashboard showing ingest volume per stream, so growth is invisible until queries break.

Best practices

• Decide labels deliberately: bounded, known in advance, used for selection.
• Put high-cardinality but searchable values in structured metadata, not labels.
• Run --analyze-labels periodically, not just when something breaks.
• Drop noise at the agent, before it costs you anything downstream.
• Emit structured logs from your applications so parsers are cheap and reliable.
• Start monolithic and move to simple scalable only when you have a reason.
• TSDB, schema v13, chunks in object storage.
• Turn retention on properly and use per-stream retention to keep costs sane.
• Order queries: select, filter, parse.
• Alert on log-derived metrics through the ruler rather than eyeballing dashboards.
• Graph ingest volume per job so cardinality growth shows up as a trend.

FAQ

Is Loki a drop-in replacement for the ELK stack?

No, and expecting it to be is where most disappointment comes from. Loki is dramatically cheaper to run and much better suited to “show me the logs for this service around this time”. Elasticsearch is better at ad-hoc full-text search across everything with no idea where to start. Pick based on which of those you do more often.

Why is my Loki so slow?

Almost always label cardinality. Run logcli series '{}' --analyze-labels and look for the outlier. The second most common cause is a query that parses before it filters. Hardware is a distant third.

Should I still use Promtail?

No. It reached end of life in March 2026 and receives no security patches. Use Grafana Alloy, and run alloy convert against your existing config rather than rewriting it from scratch.

How many labels is too many?

The count matters less than the product of their distinct values. Five labels with a handful of values each is fine. Three labels where one has fifty thousand values is not. Multiply before you add.

Can I run Loki on a single VPS?

Yes, and for a handful of services it is a perfectly good answer. Monolithic mode alongside Grafana and Prometheus fits on a modest box from any provider. Point chunk storage at an object store rather than local disk so a dead server does not take your history with it.

How do I search for a specific request across services?

Attach the trace or request ID as structured metadata at the agent, then filter on it. You get the high-cardinality lookup you wanted without creating a stream per request.

Self-host or use a managed service?

Self-hosting a monolithic Loki is genuinely low effort. Self-hosting a highly available multi-tenant Loki is a real ongoing job, and at that point Grafana Cloud or a hosted alternative deserves an honest cost comparison including your time. The break-even is further out than vendors suggest and closer than enthusiasts admit.

The one thing to remember

Labels select, filters search. Every serious problem with centralized logging with Loki traces back to someone treating a label as a search field. Loki will accept it without complaint, and the cost arrives weeks later as a query that will not finish.

So before adding any label, multiply out how many streams it creates. If the answer is more than a few thousand, it belongs in structured metadata or in the log line. And check your cardinality on a schedule, because this degrades gradually and you will not notice the day it crosses the line.

Need a logging setup that stays fast?

Most Loki installations I get asked to look at were fine on day one and got slower every week. Work I take on:

• Diagnosing a slow Loki: cardinality analysis, query shape review, and a concrete list of labels to move.
• Migrating Promtail configurations to Grafana Alloy, including Docker and Kubernetes log collection.
• Building a centralized logging stack from scratch: Loki, Alloy, Grafana, object storage, retention and dashboards.
• Restructuring labels and introducing structured metadata without losing queryability.
• Retention and storage cost work: per-stream policies, agent-side dropping, object storage tiering.
• Log-based alerting through the ruler, wired into an existing Prometheus and Alertmanager setup.

Send me the output of logcli series '{}' --analyze-labels and a query that feels slow, and I will tell you where the problem is.

The post One Label Too Many: Centralized Logging With Loki Without Wrecking It appeared first on John Nessime.