<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Observability | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/observability/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/observability/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Fri, 31 Jul 2026 22:54:35 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.0.2</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Observability | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/observability/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Metrics, Logs and Traces: What Each One Answers</title>
		<link>https://john-nessime.com/blog/devops/metrics-logs-traces/</link>
					<comments>https://john-nessime.com/blog/devops/metrics-logs-traces/#respond</comments>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 31 Jul 2026 22:54:33 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Web Performance]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[OpenTelemetry]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Tracing]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=52</guid>

					<description><![CDATA[<p>Metrics tell you that something is wrong. Traces tell you where. Logs tell you why. Use them in that order and each step narrows the search — start with logs and you are grepping a firehose. Here is what each one is actually for, and the field that connects all three.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/metrics-logs-traces/">Metrics, Logs and Traces: What Each One Answers</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Checkout is slow. Somebody says it in Slack, and within a minute three people are looking at three different screens. One is scrolling a dashboard where every panel looks normal. One is grepping logs for the word &#8220;error&#8221; and finding nothing. One is reading a trace for a request that completed in 40ms and wondering why they bothered.</p>



<p class="wp-block-paragraph">Nobody is doing anything wrong. They are just using the wrong tool for the question at hand, and none of them has said out loud what question they are trying to answer.</p>



<p class="wp-block-paragraph">Metrics, logs and traces get called the three pillars of observability, which makes them sound like three views of the same thing. They are not. They answer three different questions, and knowing which question you are asking is most of the skill.</p>



<h2 class="wp-block-heading">The one-line version</h2>



<ul class="wp-block-list">
<li><strong>Metrics</strong> tell you <em>that</em> something is wrong, and when it started.</li>

<li><strong>Traces</strong> tell you <em>where</em> in the system it is wrong.</li>

<li><strong>Logs</strong> tell you <em>why</em> that particular thing went wrong.</li>
</ul>



<p class="wp-block-paragraph">That ordering is also the order you should use them in during an incident. Starting with logs is the most common mistake, and it is why people end up grepping for twenty minutes before discovering the problem was a downstream service that never logged anything at all.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Metrics: numbers over time, and the cost of a label</h2>



<p class="wp-block-paragraph">A metric is a number sampled repeatedly. Requests per second, memory used, queue depth. They are cheap to store because each sample is a timestamp and a value, and they aggregate well — which is exactly what makes them right for dashboards and alerts, and wrong for investigating one specific user&#8217;s problem.</p>



<pre class="wp-block-code"><code># Error rate as a proportion of all requests, per service
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
  /
sum(rate(http_requests_total[5m])) by (service)</code></pre>



<p class="wp-block-paragraph">That query is answerable in milliseconds across months of history. No log search comes close.</p>



<h3 class="wp-block-heading">Averages lie, and they lie in a specific direction</h3>



<p class="wp-block-paragraph">An average response time of 200ms is compatible with 95% of requests at 50ms and 5% at three seconds. The people having a terrible time are invisible in the mean, and they are the ones who complain.</p>



<pre class="wp-block-code"><code># p99 latency by endpoint
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, endpoint)
)</code></pre>



<p class="wp-block-paragraph">Alert on p95 or p99, look at p50 to understand the typical case, and be aware that averaging percentiles across instances is mathematically meaningless — you have to aggregate the histogram buckets first, which is what the query above does.</p>



<h3 class="wp-block-heading">Cardinality is the thing that breaks Prometheus</h3>



<p class="wp-block-paragraph">Every unique combination of label values creates a separate time series. Add a label with unbounded values and you multiply your storage and memory by the number of distinct values.</p>



<pre class="wp-block-code"><code># Fine. A handful of endpoints, a handful of statuses.
http_requests_total{endpoint="/checkout", status="200"}

# Catastrophic. One time series per user, forever.
http_requests_total{endpoint="/checkout", user_id="8a3f...", request_id="..."}</code></pre>



<p class="wp-block-paragraph">User IDs, request IDs, session tokens, full URLs with parameters, email addresses — none of these belong in a metric label. They belong in logs and traces, which are built for high-cardinality data.</p>



<p class="wp-block-paragraph">To see whether you already have a problem:</p>



<pre class="wp-block-code"><code># Which metric names have the most series?
topk(10, count by (__name__)({__name__=~".+"}))</code></pre>



<p class="wp-block-paragraph">Run that on any Prometheus that has been growing unexpectedly. The answer is usually one metric somebody added a request ID to.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Logs: the detail, once you know where to look</h2>



<p class="wp-block-paragraph">Logs are discrete events with arbitrary detail attached. Their strength is precision — the actual exception, the actual SQL statement, the actual value that was null. Their weakness is volume and cost, since you are storing every event rather than an aggregate.</p>



<h3 class="wp-block-heading">Structured or it did not happen</h3>



<p class="wp-block-paragraph">Compare these two lines for the same event:</p>



<pre class="wp-block-code"><code># Human readable, machine hostile
2026-01-15 14:32:11 ERROR Payment failed for user 8a3f after 3 retries

# Structured: queryable without regex archaeology
{"ts":"2026-01-15T14:32:11Z","level":"error","msg":"payment failed",
 "user_id":"8a3f","retries":3,"provider":"stripe","error_code":"card_declined",
 "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","duration_ms":1840}</code></pre>



<p class="wp-block-paragraph">With the second, &#8220;how many payments failed with card_declined on stripe in the last hour, grouped by retry count&#8221; is a query. With the first it is a regular expression that will break the next time someone rewords the message.</p>



<h3 class="wp-block-heading">The field that ties everything together</h3>



<p class="wp-block-paragraph">Notice <code>trace_id</code> in that log line. That single field is what turns three separate tools into one workflow. Find a slow trace, take its ID, and pull every log line from every service involved in that exact request.</p>



<p class="wp-block-paragraph">If you do one thing after reading this, make it that: get trace and span IDs into your structured logs. It is usually a few lines in a logging middleware and it changes how debugging feels.</p>



<h3 class="wp-block-heading">Levels, used properly</h3>



<ul class="wp-block-list">
<li><strong>ERROR</strong> — something failed and a human may need to act. If nobody would act on it, it is not an error.</li>

<li><strong>WARN</strong> — recovered, but worth knowing. A retry succeeded, a fallback was used.</li>

<li><strong>INFO</strong> — meaningful state changes. Started, stopped, config loaded, job completed.</li>

<li><strong>DEBUG</strong> — off in production, on when you are actively investigating.</li>
</ul>



<p class="wp-block-paragraph">The failure mode is logging everything at INFO. You get volume, cost, and a signal that nobody reads because it is 99% noise. If your ERROR channel fires constantly and nobody looks, you have trained your team to ignore errors.</p>



<h3 class="wp-block-heading">Sampling, when volume gets expensive</h3>



<p class="wp-block-paragraph">Keep all errors and warnings. Sample the successful path — one in a hundred INFO lines from a healthy endpoint tells you the same thing as all hundred, at a fraction of the cost. Never sample errors. The one you drop is the one you needed.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Traces: where the time actually went</h2>



<p class="wp-block-paragraph">A trace follows one request across every service it touches. Each unit of work is a span, with a start time, duration, and a parent — so the trace becomes a tree showing exactly where the milliseconds went.</p>



<pre class="wp-block-code"><code>POST /checkout                                    1840ms
├─ auth.verify_token                                12ms
├─ cart.get_items                                   34ms
│  └─ SELECT * FROM cart_items WHERE ...            28ms
├─ inventory.check_stock                            41ms
├─ payment.charge                                 1690ms   ← here
│  ├─ HTTP POST api.stripe.com/v1/charges         1680ms
│  └─ db.insert payment_record                       8ms
└─ order.create                                     52ms</code></pre>



<p class="wp-block-paragraph">No amount of dashboard-staring gets you there that fast. The metric told you p99 checkout latency doubled; the trace tells you it is one outbound HTTP call, and now you know whether to look at your code or send an email to a vendor.</p>



<h3 class="wp-block-heading">Traces are also how you find work you did not know about</h3>



<p class="wp-block-paragraph">The classic find is the N+1 query — a trace showing four hundred nearly identical database spans where you expected one. That is invisible in metrics, because four hundred fast queries look healthy in aggregate, and invisible in logs unless you happen to log every query.</p>



<h3 class="wp-block-heading">Context propagation is the part that breaks</h3>



<p class="wp-block-paragraph">Tracing works because each service passes the trace context to the next one, usually in the W3C <code>traceparent</code> header:</p>



<pre class="wp-block-code"><code>traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  │                                │                │
             │  └─ trace ID                      └─ parent span   └─ flags
             └─ version</code></pre>



<p class="wp-block-paragraph">Break that chain anywhere and your trace splits in two, with the second half orphaned and useless. The usual culprits: a service that does not forward the header, a message queue where nobody propagated context into the consumer, or a thread pool that lost the context on handoff. Async boundaries are where this fails most often.</p>



<h3 class="wp-block-heading">Sampling, and why tail-based is worth the trouble</h3>



<p class="wp-block-paragraph">Tracing everything at scale is expensive, so you sample. The distinction matters:</p>



<ul class="wp-block-list">
<li><strong>Head-based</strong> — decide at the start of the request. Simple, cheap, and it throws away most of your errors, because errors are rare and the dice were rolled before anything went wrong.</li>

<li><strong>Tail-based</strong> — decide after the request finishes, when you know it was slow or failed. Keep every error, keep the slow tail, sample the boring successes at 1%.</li>
</ul>



<p class="wp-block-paragraph">Tail-based needs a collector that buffers spans until the trace completes, which costs memory and adds a component. It is almost always worth it, because a trace store full of successful requests is a trace store you never open.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Using all three: an incident, in order</h2>



<ol class="wp-block-list">
<li><strong>An alert fires on a metric.</strong> Checkout p99 latency crossed its threshold at 14:20. You know what and when.</li>

<li><strong>Check the metric&#8217;s shape.</strong> Step change or gradual climb? Correlated with a deploy, traffic spike, or nothing? Which service, which endpoint?</li>

<li><strong>Open a slow trace from that window.</strong> Not an average one — filter for the slow tail. Find the span holding the time.</li>

<li><strong>Take the trace ID into your logs.</strong> Now you are reading a handful of lines from the exact request, not grepping a firehose.</li>

<li><strong>Confirm the fix with the metric</strong> you started from. The graph coming back down is what closes the incident.</li>
</ol>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Every step narrows the search. Metrics cover everything at low detail, traces cover one request at high detail, logs cover one moment at full detail. Going straight to logs means searching everything at full detail, which is why it feels like archaeology.</p>
</blockquote>



<h2 class="wp-block-heading">What to instrument if you are starting from nothing</h2>



<p class="wp-block-paragraph">Do not try to do all three at once. In order of value per hour spent:</p>



<ol class="wp-block-list">
<li><strong>Request rate, error rate and duration per endpoint.</strong> These three metrics answer most operational questions and take an afternoon with a middleware.</li>

<li><strong>Structured logs with a request ID.</strong> Even without tracing, a request ID threaded through your logs is transformative.</li>

<li><strong>Host metrics.</strong> CPU, memory, disk, network. Answers &#8220;is the machine underneath healthy&#8221; before you debug the application.</li>

<li><strong>Tracing on your slowest or most critical path.</strong> Not everything. Checkout, or whatever your equivalent is.</li>

<li><strong>Trace IDs in log lines,</strong> once tracing exists. This is the step that connects the tools.</li>
</ol>



<p class="wp-block-paragraph">Instrument with OpenTelemetry rather than a vendor SDK. The API is vendor-neutral, so switching backends later is a collector config change instead of touching every service. That is the single decision most likely to save you a painful migration.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Starting an investigation in logs rather than metrics.</li>

<li>Putting user IDs, request IDs or full URLs into metric labels.</li>

<li>Alerting on averages instead of percentiles.</li>

<li>Averaging percentiles across instances, which is not a meaningful operation.</li>

<li>Unstructured log messages that require regular expressions to query.</li>

<li>Head-based trace sampling, which discards most of your errors by construction.</li>

<li>Sampling error logs to save money.</li>

<li>No trace or request ID in logs, leaving the three tools unconnected.</li>

<li>Instrumenting with a vendor SDK and discovering the cost of that choice at renewal time.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep metric cardinality bounded and check your top series periodically.</li>

<li>Log structured JSON with consistent field names across services.</li>

<li>Propagate trace context everywhere, especially across queues and async boundaries.</li>

<li>Use tail-based sampling so errors and slow requests are always kept.</li>

<li>Set retention by usefulness: metrics for months, traces for days, logs somewhere between.</li>

<li>Alert on symptoms users feel — latency, errors, saturation — not on internal counters nobody can act on.</li>

<li>Standardise on OpenTelemetry so the backend stays replaceable.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need all three?</h3>



<p class="wp-block-paragraph">Not immediately. A single service with good metrics and structured logs is well covered. Tracing earns its cost when a request crosses several services and &#8220;which one is slow&#8221; stops being obvious.</p>



<h3 class="wp-block-heading">What is the difference between monitoring and observability?</h3>



<p class="wp-block-paragraph">Monitoring watches for failure modes you predicted. Observability is being able to answer questions you did not think of in advance. Metrics and alerts are mostly monitoring; high-cardinality traces and structured logs are what make a system observable.</p>



<h3 class="wp-block-heading">Why is my Prometheus using so much memory?</h3>



<p class="wp-block-paragraph">Almost always cardinality. Memory tracks the number of active time series, not the number of hosts. Find the offending metric with a top-series query — it will usually be one where somebody added an unbounded label.</p>



<h3 class="wp-block-heading">Can I put logs into metrics or the reverse?</h3>



<p class="wp-block-paragraph">You can derive metrics from logs, and it is a reasonable stopgap when you cannot change the application. It is more expensive than emitting metrics directly and the resolution is worse. Going the other way — reconstructing event detail from metrics — is not possible, since aggregation discards it.</p>



<h3 class="wp-block-heading">How long should I keep each?</h3>



<p class="wp-block-paragraph">Metrics are cheap and useful for capacity planning, so months. Traces are large and mostly interesting while an incident is fresh, so days. Logs sit in between and are usually driven by compliance rather than debugging value.</p>



<h3 class="wp-block-heading">Is tracing worth it for a monolith?</h3>



<p class="wp-block-paragraph">Often yes, because spans still reveal internal structure — which query, which external call, how many times. The N+1 problem shows up just as clearly in a monolith as in a distributed system.</p>



<h2 class="wp-block-heading">Wrapping up</h2>



<p class="wp-block-paragraph">Metrics, logs and traces are not three ways of looking at the same data. Metrics are aggregates over time and answer whether something is wrong. Traces follow one request and answer where. Logs are individual events and answer why. Reach for them in that order and each step makes the next one smaller.</p>



<p class="wp-block-paragraph">If your setup is incomplete, the highest-value gap is usually the connective tissue rather than another data source. Trace IDs in structured logs, consistent field names, bounded metric labels. Those three make the tools you already have work together, which is worth more than adding a fourth.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Want your observability stack to actually answer questions?</h2>



<p class="wp-block-paragraph">I work with teams on observability that earns its cost — metrics with sane cardinality, structured logging that is queryable, OpenTelemetry instrumentation that survives a backend change, and alerts that fire on things people can act on.</p>



<p class="wp-block-paragraph">If your dashboards look healthy while users complain, you can hire me on Upwork.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/metrics-logs-traces/">Metrics, Logs and Traces: What Each One Answers</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://john-nessime.com/blog/devops/metrics-logs-traces/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>Prometheus + Grafana From Scratch on One Server</title>
		<link>https://john-nessime.com/blog/devops/prometheus-grafana-one-server/</link>
					<comments>https://john-nessime.com/blog/devops/prometheus-grafana-one-server/#respond</comments>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 30 Jul 2026 15:23:39 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=39</guid>

					<description><![CDATA[<p>Two static binaries, a package and four systemd units. How to install Prometheus and Grafana on a single server from scratch, wire in node_exporter, put nginx and TLS in front, and keep the ports that have no authentication off the public internet.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/prometheus-grafana-one-server/">Prometheus + Grafana From Scratch on One Server</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">You want to know when the disk fills up before a customer tells you. Search for how to do that and you get pointed at a hosted platform with per-host pricing, or a twelve-service Docker Compose file that somebody copy-pasted from a blog and does not fully understand either.</p>



<p class="wp-block-paragraph">For a single server, neither is the right answer. Prometheus and Grafana are two static Go binaries and a package. They run happily under systemd, use very little memory at this scale, and when something breaks you can read the logs and actually fix it. No container networking to debug, no orchestration layer between you and the problem.</p>



<p class="wp-block-paragraph">This walks through the whole thing: install Prometheus and Grafana from scratch, wire in node_exporter, put the lot behind nginx with TLS, and lock down the ports that should never have been public in the first place. Commands work on both Debian-family and RHEL-family systems, with the differences called out.</p>



<h2 class="wp-block-heading">What you are building</h2>



<p class="wp-block-paragraph">Four moving parts, all on one box:</p>



<ul class="wp-block-list">
<li><strong>node_exporter</strong> exposes machine metrics — CPU, memory, disk, network — on port 9100.</li>

<li><strong>Prometheus</strong> scrapes that endpoint on a schedule and stores the samples in its own time series database on port 9090.</li>

<li><strong>Grafana</strong> queries Prometheus and draws the dashboards on port 3000.</li>

<li><strong>nginx</strong> terminates TLS and is the only thing listening on a public interface.</li>
</ul>



<p class="wp-block-paragraph">The critical design decision is in that last line. Prometheus and node_exporter ship with no authentication at all. Bound to a public interface, they hand your entire infrastructure inventory to anyone who runs a port scan. Everything binds to 127.0.0.1 and nginx is the single front door.</p>



<p class="wp-block-paragraph">Resource-wise, a 2 GB VPS handles this comfortably for one server&#8217;s worth of metrics. Prometheus is the memory-hungry one, and its appetite scales with the number of active time series rather than the number of hosts.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Step 1: Users and directories</h2>



<p class="wp-block-paragraph">Neither service should run as root, and neither needs a login shell.</p>



<pre class="wp-block-code"><code>sudo useradd --system --no-create-home --shell /sbin/nologin prometheus
sudo useradd --system --no-create-home --shell /sbin/nologin node_exporter

sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus</code></pre>



<p class="wp-block-paragraph">On Debian and Ubuntu the nologin shell lives at <code>/usr/sbin/nologin</code> rather than <code>/sbin/nologin</code>. Check with <code>which nologin</code> if useradd complains.</p>



<h2 class="wp-block-heading">Step 2: Install Prometheus</h2>



<p class="wp-block-paragraph">Prometheus ships as a single static binary. Rather than hardcoding a version that will be stale next month, pull the current release tag from the API:</p>



<pre class="wp-block-code"><code>PROM_VERSION=$(curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest 
  | grep -Po '"tag_name": "vK[^"]*')
echo "Installing Prometheus ${PROM_VERSION}"

cd /tmp
curl -LO "https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
tar xzf "prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
cd "prometheus-${PROM_VERSION}.linux-amd64"

sudo cp prometheus promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool</code></pre>



<p class="wp-block-paragraph">If you are on ARM, swap <code>linux-amd64</code> for <code>linux-arm64</code>.</p>



<h3 class="wp-block-heading">The configuration file</h3>



<p class="wp-block-paragraph">Write <code>/etc/prometheus/prometheus.yml</code>:</p>



<pre class="wp-block-code"><code>global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["127.0.0.1:9090"]

  - job_name: node
    static_configs:
      - targets: ["127.0.0.1:9100"]
        labels:
          instance: "web-01"</code></pre>



<p class="wp-block-paragraph">Set that <code>instance</code> label deliberately. It follows every metric into every dashboard and alert, and renaming it later means your historical graphs split in two.</p>



<p class="wp-block-paragraph">Validate before you start anything:</p>



<pre class="wp-block-code"><code>promtool check config /etc/prometheus/prometheus.yml</code></pre>



<h3 class="wp-block-heading">The systemd unit</h3>



<p class="wp-block-paragraph">Write <code>/etc/systemd/system/prometheus.service</code>:</p>



<pre class="wp-block-code"><code>[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/prometheus 
  --config.file=/etc/prometheus/prometheus.yml 
  --storage.tsdb.path=/var/lib/prometheus 
  --storage.tsdb.retention.time=30d 
  --storage.tsdb.retention.size=8GB 
  --web.listen-address=127.0.0.1:9090
ExecReload=/bin/kill -HUP $MAINPID

NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/prometheus
PrivateTmp=true

[Install]
WantedBy=multi-user.target</code></pre>



<p class="wp-block-paragraph">Two things worth noticing. <code>--web.listen-address=127.0.0.1:9090</code> is what keeps Prometheus off the public internet. And setting <em>both</em> retention flags matters — time-based retention alone will not save you if metric volume spikes, because Prometheus honours whichever limit it hits first.</p>



<pre class="wp-block-code"><code>sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheus</code></pre>



<h2 class="wp-block-heading">Step 3: node_exporter</h2>



<p class="wp-block-paragraph">Same pattern — a static binary and a unit file.</p>



<pre class="wp-block-code"><code>NE_VERSION=$(curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest 
  | grep -Po '"tag_name": "vK[^"]*')

cd /tmp
curl -LO "https://github.com/prometheus/node_exporter/releases/download/v${NE_VERSION}/node_exporter-${NE_VERSION}.linux-amd64.tar.gz"
tar xzf "node_exporter-${NE_VERSION}.linux-amd64.tar.gz"
sudo cp "node_exporter-${NE_VERSION}.linux-amd64/node_exporter" /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter</code></pre>



<p class="wp-block-paragraph">Then <code>/etc/systemd/system/node_exporter.service</code>:</p>



<pre class="wp-block-code"><code>[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/node_exporter 
  --web.listen-address=127.0.0.1:9100

NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target</code></pre>



<pre class="wp-block-code"><code>sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

# Confirm it is actually producing metrics
curl -s http://127.0.0.1:9100/metrics | head -20</code></pre>



<p class="wp-block-paragraph">Now open <code>http://127.0.0.1:9090/targets</code> — via an SSH tunnel, since it is not public — and both jobs should read UP.</p>



<pre class="wp-block-code"><code># From your laptop
ssh -L 9090:127.0.0.1:9090 you@your-server</code></pre>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Step 4: Install Grafana</h2>



<p class="wp-block-paragraph">Grafana is packaged properly, so use the official repository rather than a tarball. It gets you signed updates through your normal patching process.</p>



<p class="wp-block-paragraph"><strong>Debian and Ubuntu:</strong></p>



<pre class="wp-block-code"><code>sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key 
  | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg &gt; /dev/null

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" 
  | sudo tee /etc/apt/sources.list.d/grafana.list

sudo apt-get update
sudo apt-get install -y grafana</code></pre>



<p class="wp-block-paragraph"><strong>RHEL, AlmaLinux, Rocky:</strong></p>



<pre class="wp-block-code"><code>sudo tee /etc/yum.repos.d/grafana.repo &gt; /dev/null &lt;&lt;'EOF'
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

sudo dnf install -y grafana</code></pre>



<h3 class="wp-block-heading">Bind it to localhost too</h3>



<p class="wp-block-paragraph">Edit <code>/etc/grafana/grafana.ini</code>:</p>



<pre class="wp-block-code"><code>[server]
http_addr = 127.0.0.1
http_port = 3000
domain = metrics.example.com
root_url = https://metrics.example.com/

[security]
admin_user = admin
admin_password = put-something-real-here
cookie_secure = true

[users]
allow_sign_up = false</code></pre>



<p class="wp-block-paragraph">Getting <code>root_url</code> right matters more than it looks. Set it wrong and Grafana generates broken links and redirects behind the proxy, which produces a confusing half-working UI rather than a clean error.</p>



<pre class="wp-block-code"><code>sudo systemctl enable --now grafana-server</code></pre>



<h2 class="wp-block-heading">Step 5: Put nginx and TLS in front</h2>



<p class="wp-block-paragraph">This is the only piece that faces the internet.</p>



<pre class="wp-block-code"><code>server {
    listen 443 ssl;
    server_name metrics.example.com;

    ssl_certificate     /etc/letsencrypt/live/metrics.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/metrics.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Grafana Live needs websockets, and breaks silently without this
    location /api/live/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade    $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host       $host;
        proxy_pass http://127.0.0.1:3000;
    }
}</code></pre>



<p class="wp-block-paragraph">Note <code>fullchain.pem</code> rather than <code>cert.pem</code>. Getting that wrong produces a site that works in your browser and fails for anything else.</p>



<p class="wp-block-paragraph">Then close the metrics ports at the firewall, belt and braces:</p>



<pre class="wp-block-code"><code># firewalld
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

# ufw
sudo ufw allow 443/tcp

# Verify nothing unexpected is listening publicly
sudo ss -tlnp | grep -E '9090|9100|3000'</code></pre>



<p class="wp-block-paragraph">Every line of that last command should show <code>127.0.0.1</code>. If you see <code>0.0.0.0</code> or <code>*</code>, something is exposed and you should fix it before going any further.</p>



<h2 class="wp-block-heading">Step 6: Connect Grafana to Prometheus</h2>



<p class="wp-block-paragraph">You can click through the UI, but provisioning it as a file means a rebuilt server comes back identical. Create <code>/etc/grafana/provisioning/datasources/prometheus.yml</code>:</p>



<pre class="wp-block-code"><code>apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://127.0.0.1:9090
    isDefault: true</code></pre>



<p class="wp-block-paragraph">Restart Grafana, log in, and import dashboard ID <strong>1860</strong> from the Grafana dashboard library — Node Exporter Full. It is comprehensive, well maintained, and will show you more about your server than you knew was measurable.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Sizing the disk before it bites</h2>



<p class="wp-block-paragraph">Prometheus storage is predictable. Roughly:</p>



<pre class="wp-block-code"><code>disk = retention_seconds × samples_per_second × bytes_per_sample</code></pre>



<p class="wp-block-paragraph">Compressed samples land in the region of one to two bytes each. Measure your actual rate rather than guessing — run this in the Prometheus query box:</p>



<pre class="wp-block-code"><code>rate(prometheus_tsdb_head_samples_appended_total[5m])</code></pre>



<p class="wp-block-paragraph">One server with node_exporter at a 15 second interval is a small number, and 30 days fits in a few gigabytes. The number climbs fast once you add exporters, so keep <code>--storage.tsdb.retention.size</code> set as a hard ceiling. Prometheus filling the root partition takes the rest of the box down with it, which is a memorable way to discover your monitoring has no monitoring.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">A target shows DOWN</h3>



<p class="wp-block-paragraph">Work outward from the exporter:</p>



<pre class="wp-block-code"><code>curl -s http://127.0.0.1:9100/metrics | head
sudo systemctl status node_exporter
sudo journalctl -u prometheus -n 50 --no-pager</code></pre>



<p class="wp-block-paragraph">On RHEL-family systems with SELinux enforcing, this is frequently SELinux rather than your config. Check before you start changing things:</p>



<pre class="wp-block-code"><code>sudo ausearch -m avc -ts recent</code></pre>



<p class="wp-block-paragraph">If that shows denials, write a policy for them. Do not reach for <code>setenforce 0</code> — it fixes today&#8217;s symptom and quietly removes a layer of protection from the whole machine.</p>



<h3 class="wp-block-heading">Grafana loads but every panel says &#8220;No data&#8221;</h3>



<p class="wp-block-paragraph">Almost always the datasource URL. It must be <code>http://127.0.0.1:9090</code> from Grafana&#8217;s point of view, not your public hostname. Test the datasource from its settings page — Grafana reports the actual connection error, which is more useful than the empty panel.</p>



<h3 class="wp-block-heading">Prometheus will not start</h3>



<pre class="wp-block-code"><code>sudo journalctl -u prometheus -n 50 --no-pager
promtool check config /etc/prometheus/prometheus.yml
ls -la /var/lib/prometheus</code></pre>



<p class="wp-block-paragraph">The usual culprits are a YAML indentation error, or <code>/var/lib/prometheus</code> not being owned by the prometheus user after a manual file copy.</p>



<h3 class="wp-block-heading">Broken styling or redirect loops behind the proxy</h3>



<p class="wp-block-paragraph"><code>root_url</code> in grafana.ini does not match how you are actually reaching the site. If you are serving from a subpath, you also need <code>serve_from_sub_path = true</code>.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Leaving 9090, 9100 or 3000 on a public interface. Prometheus has no authentication whatsoever.</li>

<li>Keeping the default Grafana admin password because it prompts you to change it and you clicked past.</li>

<li>Setting no retention limits and discovering the problem when the root partition fills.</li>

<li>Dropping <code>scrape_interval</code> to 1s because more data seems better. It multiplies storage and buys you nothing at this scale.</li>

<li>Never backing up <code>/var/lib/grafana/grafana.db</code>, which holds every dashboard you have built.</li>

<li>Running everything as root because the tutorial you followed did.</li>

<li>Building dashboards but configuring no alerts, so you still find out from a user.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Bind every component to localhost and expose exactly one TLS endpoint.</li>

<li>Set both time and size retention. Whichever triggers first is the one you want.</li>

<li>Provision datasources and dashboards as files, so the config lives in version control.</li>

<li>Back up the Grafana SQLite database on the same schedule as everything else that matters.</li>

<li>Add an external uptime check. Self-hosted monitoring cannot tell you that the server it lives on has gone.</li>

<li>Write a handful of alerts you will genuinely act on. Alerts nobody responds to train people to ignore alerts.</li>

<li>Use systemd&#8217;s sandboxing directives. They cost nothing and limit what a compromised exporter can reach.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Should I use Docker instead?</h3>



<p class="wp-block-paragraph">For a single server, systemd is simpler. Two static binaries and a package, with no container networking layer between you and the logs. Containers earn their keep once you are managing several hosts or already run everything else that way.</p>



<h3 class="wp-block-heading">How much RAM does this need?</h3>



<p class="wp-block-paragraph">A 2 GB VPS is comfortable for one server. Prometheus memory tracks active time series rather than host count, so it grows when you add exporters, not when you add machines with the same exporters.</p>



<h3 class="wp-block-heading">Can I monitor more servers later?</h3>



<p class="wp-block-paragraph">Yes. Install node_exporter on each and add them as targets. The moment metrics cross a network, though, that traffic needs protecting — a VPN, an SSH tunnel, or TLS with client certificates. Do not simply open 9100 to the world.</p>



<h3 class="wp-block-heading">Where do alerts come from?</h3>



<p class="wp-block-paragraph">Two options. Grafana&#8217;s built-in alerting works from dashboard queries and is the quicker path. Alertmanager, run alongside Prometheus, is more powerful for routing, grouping and silencing. Start with Grafana&#8217;s and move if you outgrow it.</p>



<h3 class="wp-block-heading">How long should I keep metrics?</h3>



<p class="wp-block-paragraph">Thirty days covers most troubleshooting and capacity questions. Longer retention on a single box is usually a sign you want a remote-write backend rather than a bigger disk.</p>



<h3 class="wp-block-heading">Do I need node_exporter if I only care about my application?</h3>



<p class="wp-block-paragraph">Install it anyway. When the application misbehaves, the first question is whether the machine underneath it is healthy, and node_exporter is what answers that.</p>



<h2 class="wp-block-heading">Wrapping up</h2>



<p class="wp-block-paragraph">Once you install Prometheus and Grafana this way, the whole stack is four systemd units and three config files. Everything binds to localhost, nginx handles TLS, and there is no layer of abstraction between you and a problem when something breaks at an inconvenient hour.</p>



<p class="wp-block-paragraph">Get the dashboard working first, then spend an evening writing three or four alerts you will actually respond to. A dashboard tells you what happened when you go looking. An alert is what stops you needing to look.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Want this set up properly?</h2>



<p class="wp-block-paragraph">I build monitoring stacks for teams who would rather not spend a weekend on it — Prometheus and Grafana on bare servers or in Kubernetes, exporters for the things you actually run, alert rules tuned so they mean something, and dashboards that answer real questions rather than filling a screen.</p>



<p class="wp-block-paragraph">If that sounds useful, you can hire me on Upwork.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/prometheus-grafana-one-server/">Prometheus + Grafana From Scratch on One Server</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://john-nessime.com/blog/devops/prometheus-grafana-one-server/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
	</channel>
</rss>
