Checkout is slow. Somebody says it in Slack, and within a minute three people are looking at three different screens. One is scrolling a dashboard where every panel looks normal. One is grepping logs for the word “error” and finding nothing. One is reading a trace for a request that completed in 40ms and wondering why they bothered.<\/p>\n\n\n\n
Nobody is doing anything wrong. They are just using the wrong tool for the question at hand, and none of them has said out loud what question they are trying to answer.<\/p>\n\n\n\n
Metrics, logs and traces get called the three pillars of observability, which makes them sound like three views of the same thing. They are not. They answer three different questions, and knowing which question you are asking is most of the skill.<\/p>\n\n\n\n
That ordering is also the order you should use them in during an incident. Starting with logs is the most common mistake, and it is why people end up grepping for twenty minutes before discovering the problem was a downstream service that never logged anything at all.<\/p>\n\n\n\n
A metric is a number sampled repeatedly. Requests per second, memory used, queue depth. They are cheap to store because each sample is a timestamp and a value, and they aggregate well \u2014 which is exactly what makes them right for dashboards and alerts, and wrong for investigating one specific user’s problem.<\/p>\n\n\n\n
# Error rate as a proportion of all requests, per service\nsum(rate(http_requests_total{status=~\"5..\"}[5m])) by (service)\n \/\nsum(rate(http_requests_total[5m])) by (service)<\/code><\/pre>\n\n\n\nThat query is answerable in milliseconds across months of history. No log search comes close.<\/p>\n\n\n\n
Averages lie, and they lie in a specific direction<\/h3>\n\n\n\n
An average response time of 200ms is compatible with 95% of requests at 50ms and 5% at three seconds. The people having a terrible time are invisible in the mean, and they are the ones who complain.<\/p>\n\n\n\n
# p99 latency by endpoint\nhistogram_quantile(0.99,\n sum(rate(http_request_duration_seconds_bucket[5m])) by (le, endpoint)\n)<\/code><\/pre>\n\n\n\nAlert on p95 or p99, look at p50 to understand the typical case, and be aware that averaging percentiles across instances is mathematically meaningless \u2014 you have to aggregate the histogram buckets first, which is what the query above does.<\/p>\n\n\n\n
Cardinality is the thing that breaks Prometheus<\/h3>\n\n\n\n
Every unique combination of label values creates a separate time series. Add a label with unbounded values and you multiply your storage and memory by the number of distinct values.<\/p>\n\n\n\n
# Fine. A handful of endpoints, a handful of statuses.\nhttp_requests_total{endpoint=\"\/checkout\", status=\"200\"}\n\n# Catastrophic. One time series per user, forever.\nhttp_requests_total{endpoint=\"\/checkout\", user_id=\"8a3f...\", request_id=\"...\"}<\/code><\/pre>\n\n\n\nUser IDs, request IDs, session tokens, full URLs with parameters, email addresses \u2014 none of these belong in a metric label. They belong in logs and traces, which are built for high-cardinality data.<\/p>\n\n\n\n
To see whether you already have a problem:<\/p>\n\n\n\n
# Which metric names have the most series?\ntopk(10, count by (__name__)({__name__=~\".+\"}))<\/code><\/pre>\n\n\n\nRun that on any Prometheus that has been growing unexpectedly. The answer is usually one metric somebody added a request ID to.<\/p>\n\n\n\n
\n\n\n\nLogs: the detail, once you know where to look<\/h2>\n\n\n\n
Logs are discrete events with arbitrary detail attached. Their strength is precision \u2014 the actual exception, the actual SQL statement, the actual value that was null. Their weakness is volume and cost, since you are storing every event rather than an aggregate.<\/p>\n\n\n\n
Structured or it did not happen<\/h3>\n\n\n\n
Compare these two lines for the same event:<\/p>\n\n\n\n
# Human readable, machine hostile\n2026-01-15 14:32:11 ERROR Payment failed for user 8a3f after 3 retries\n\n# Structured: queryable without regex archaeology\n{\"ts\":\"2026-01-15T14:32:11Z\",\"level\":\"error\",\"msg\":\"payment failed\",\n \"user_id\":\"8a3f\",\"retries\":3,\"provider\":\"stripe\",\"error_code\":\"card_declined\",\n \"trace_id\":\"4bf92f3577b34da6a3ce929d0e0e4736\",\"duration_ms\":1840}<\/code><\/pre>\n\n\n\nWith the second, “how many payments failed with card_declined on stripe in the last hour, grouped by retry count” is a query. With the first it is a regular expression that will break the next time someone rewords the message.<\/p>\n\n\n\n
The field that ties everything together<\/h3>\n\n\n\n
Notice trace_id<\/code> in that log line. That single field is what turns three separate tools into one workflow. Find a slow trace, take its ID, and pull every log line from every service involved in that exact request.<\/p>\n\n\n\nIf you do one thing after reading this, make it that: get trace and span IDs into your structured logs. It is usually a few lines in a logging middleware and it changes how debugging feels.<\/p>\n\n\n\n
Levels, used properly<\/h3>\n\n\n\n\n- ERROR<\/strong> \u2014 something failed and a human may need to act. If nobody would act on it, it is not an error.<\/li>\n\n
- WARN<\/strong> \u2014 recovered, but worth knowing. A retry succeeded, a fallback was used.<\/li>\n\n
- INFO<\/strong> \u2014 meaningful state changes. Started, stopped, config loaded, job completed.<\/li>\n\n
- DEBUG<\/strong> \u2014 off in production, on when you are actively investigating.<\/li>\n<\/ul>\n\n\n\n
The failure mode is logging everything at INFO. You get volume, cost, and a signal that nobody reads because it is 99% noise. If your ERROR channel fires constantly and nobody looks, you have trained your team to ignore errors.<\/p>\n\n\n\n
Sampling, when volume gets expensive<\/h3>\n\n\n\n
Keep all errors and warnings. Sample the successful path \u2014 one in a hundred INFO lines from a healthy endpoint tells you the same thing as all hundred, at a fraction of the cost. Never sample errors. The one you drop is the one you needed.<\/p>\n\n\n\n
\n\n\n\nTraces: where the time actually went<\/h2>\n\n\n\n
A trace follows one request across every service it touches. Each unit of work is a span, with a start time, duration, and a parent \u2014 so the trace becomes a tree showing exactly where the milliseconds went.<\/p>\n\n\n\n
POST \/checkout 1840ms\n\u251c\u2500 auth.verify_token 12ms\n\u251c\u2500 cart.get_items 34ms\n\u2502 \u2514\u2500 SELECT * FROM cart_items WHERE ... 28ms\n\u251c\u2500 inventory.check_stock 41ms\n\u251c\u2500 payment.charge 1690ms \u2190 here\n\u2502 \u251c\u2500 HTTP POST api.stripe.com\/v1\/charges 1680ms\n\u2502 \u2514\u2500 db.insert payment_record 8ms\n\u2514\u2500 order.create 52ms<\/code><\/pre>\n\n\n\nNo amount of dashboard-staring gets you there that fast. The metric told you p99 checkout latency doubled; the trace tells you it is one outbound HTTP call, and now you know whether to look at your code or send an email to a vendor.<\/p>\n\n\n\n
Traces are also how you find work you did not know about<\/h3>\n\n\n\n
The classic find is the N+1 query \u2014 a trace showing four hundred nearly identical database spans where you expected one. That is invisible in metrics, because four hundred fast queries look healthy in aggregate, and invisible in logs unless you happen to log every query.<\/p>\n\n\n\n
Context propagation is the part that breaks<\/h3>\n\n\n\n
Tracing works because each service passes the trace context to the next one, usually in the W3C traceparent<\/code> header:<\/p>\n\n\n\ntraceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01\n \u2502 \u2502 \u2502 \u2502\n \u2502 \u2514\u2500 trace ID \u2514\u2500 parent span \u2514\u2500 flags\n \u2514\u2500 version<\/code><\/pre>\n\n\n\nBreak that chain anywhere and your trace splits in two, with the second half orphaned and useless. The usual culprits: a service that does not forward the header, a message queue where nobody propagated context into the consumer, or a thread pool that lost the context on handoff. Async boundaries are where this fails most often.<\/p>\n\n\n\n
Sampling, and why tail-based is worth the trouble<\/h3>\n\n\n\n
Tracing everything at scale is expensive, so you sample. The distinction matters:<\/p>\n\n\n\n
\n- Head-based<\/strong> \u2014 decide at the start of the request. Simple, cheap, and it throws away most of your errors, because errors are rare and the dice were rolled before anything went wrong.<\/li>\n\n
- Tail-based<\/strong> \u2014 decide after the request finishes, when you know it was slow or failed. Keep every error, keep the slow tail, sample the boring successes at 1%.<\/li>\n<\/ul>\n\n\n\n
Tail-based needs a collector that buffers spans until the trace completes, which costs memory and adds a component. It is almost always worth it, because a trace store full of successful requests is a trace store you never open.<\/p>\n\n\n\n
\n\n\n\nUsing all three: an incident, in order<\/h2>\n\n\n\n\n- An alert fires on a metric.<\/strong> Checkout p99 latency crossed its threshold at 14:20. You know what and when.<\/li>\n\n
- Check the metric’s shape.<\/strong> Step change or gradual climb? Correlated with a deploy, traffic spike, or nothing? Which service, which endpoint?<\/li>\n\n
- Open a slow trace from that window.<\/strong> Not an average one \u2014 filter for the slow tail. Find the span holding the time.<\/li>\n\n
- Take the trace ID into your logs.<\/strong> Now you are reading a handful of lines from the exact request, not grepping a firehose.<\/li>\n\n
- Confirm the fix with the metric<\/strong> you started from. The graph coming back down is what closes the incident.<\/li>\n<\/ol>\n\n\n\n
\nEvery step narrows the search. Metrics cover everything at low detail, traces cover one request at high detail, logs cover one moment at full detail. Going straight to logs means searching everything at full detail, which is why it feels like archaeology.<\/p>\n<\/blockquote>\n\n\n\n
What to instrument if you are starting from nothing<\/h2>\n\n\n\n
Do not try to do all three at once. In order of value per hour spent:<\/p>\n\n\n\n
\n- Request rate, error rate and duration per endpoint.<\/strong> These three metrics answer most operational questions and take an afternoon with a middleware.<\/li>\n\n
- Structured logs with a request ID.<\/strong> Even without tracing, a request ID threaded through your logs is transformative.<\/li>\n\n
- Host metrics.<\/strong> CPU, memory, disk, network. Answers “is the machine underneath healthy” before you debug the application.<\/li>\n\n
- Tracing on your slowest or most critical path.<\/strong> Not everything. Checkout, or whatever your equivalent is.<\/li>\n\n
- Trace IDs in log lines,<\/strong> once tracing exists. This is the step that connects the tools.<\/li>\n<\/ol>\n\n\n\n
Instrument with OpenTelemetry rather than a vendor SDK. The API is vendor-neutral, so switching backends later is a collector config change instead of touching every service. That is the single decision most likely to save you a painful migration.<\/p>\n\n\n\n
Common mistakes<\/h2>\n\n\n\n\n- Starting an investigation in logs rather than metrics.<\/li>\n\n
- Putting user IDs, request IDs or full URLs into metric labels.<\/li>\n\n
- Alerting on averages instead of percentiles.<\/li>\n\n
- Averaging percentiles across instances, which is not a meaningful operation.<\/li>\n\n
- Unstructured log messages that require regular expressions to query.<\/li>\n\n
- Head-based trace sampling, which discards most of your errors by construction.<\/li>\n\n
- Sampling error logs to save money.<\/li>\n\n
- No trace or request ID in logs, leaving the three tools unconnected.<\/li>\n\n
- Instrumenting with a vendor SDK and discovering the cost of that choice at renewal time.<\/li>\n<\/ul>\n\n\n\n
Best practices<\/h2>\n\n\n\n\n- Keep metric cardinality bounded and check your top series periodically.<\/li>\n\n
- Log structured JSON with consistent field names across services.<\/li>\n\n
- Propagate trace context everywhere, especially across queues and async boundaries.<\/li>\n\n
- Use tail-based sampling so errors and slow requests are always kept.<\/li>\n\n
- Set retention by usefulness: metrics for months, traces for days, logs somewhere between.<\/li>\n\n
- Alert on symptoms users feel \u2014 latency, errors, saturation \u2014 not on internal counters nobody can act on.<\/li>\n\n
- Standardise on OpenTelemetry so the backend stays replaceable.<\/li>\n<\/ul>\n\n\n\n
Frequently asked questions<\/h2>\n\n\n\nDo I need all three?<\/h3>\n\n\n\n
Not immediately. A single service with good metrics and structured logs is well covered. Tracing earns its cost when a request crosses several services and “which one is slow” stops being obvious.<\/p>\n\n\n\n
What is the difference between monitoring and observability?<\/h3>\n\n\n\n
Monitoring watches for failure modes you predicted. Observability is being able to answer questions you did not think of in advance. Metrics and alerts are mostly monitoring; high-cardinality traces and structured logs are what make a system observable.<\/p>\n\n\n\n
Why is my Prometheus using so much memory?<\/h3>\n\n\n\n
Almost always cardinality. Memory tracks the number of active time series, not the number of hosts. Find the offending metric with a top-series query \u2014 it will usually be one where somebody added an unbounded label.<\/p>\n\n\n\n
Can I put logs into metrics or the reverse?<\/h3>\n\n\n\n
You can derive metrics from logs, and it is a reasonable stopgap when you cannot change the application. It is more expensive than emitting metrics directly and the resolution is worse. Going the other way \u2014 reconstructing event detail from metrics \u2014 is not possible, since aggregation discards it.<\/p>\n\n\n\n
How long should I keep each?<\/h3>\n\n\n\n
Metrics are cheap and useful for capacity planning, so months. Traces are large and mostly interesting while an incident is fresh, so days. Logs sit in between and are usually driven by compliance rather than debugging value.<\/p>\n\n\n\n
Is tracing worth it for a monolith?<\/h3>\n\n\n\n
Often yes, because spans still reveal internal structure \u2014 which query, which external call, how many times. The N+1 problem shows up just as clearly in a monolith as in a distributed system.<\/p>\n\n\n\n
Wrapping up<\/h2>\n\n\n\n
Metrics, logs and traces are not three ways of looking at the same data. Metrics are aggregates over time and answer whether something is wrong. Traces follow one request and answer where. Logs are individual events and answer why. Reach for them in that order and each step makes the next one smaller.<\/p>\n\n\n\n
If your setup is incomplete, the highest-value gap is usually the connective tissue rather than another data source. Trace IDs in structured logs, consistent field names, bounded metric labels. Those three make the tools you already have work together, which is worth more than adding a fourth.<\/p>\n\n\n\n
\n\n\n\nWant your observability stack to actually answer questions?<\/h2>\n\n\n\n
I work with teams on observability that earns its cost \u2014 metrics with sane cardinality, structured logging that is queryable, OpenTelemetry instrumentation that survives a backend change, and alerts that fire on things people can act on.<\/p>\n\n\n\n
If your dashboards look healthy while users complain, you can hire me on Upwork.<\/p>\n\n\n\n