<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>journalctl | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/journalctl/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/journalctl/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 12:57:30 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>journalctl | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/journalctl/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Cron vs systemd Timers: The Failure Modes That Decide It</title>
		<link>https://john-nessime.com/blog/devops/cron-vs-systemd-timers/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 28 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[anacron]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Cron]]></category>
		<category><![CDATA[Crontab]]></category>
		<category><![CDATA[flock]]></category>
		<category><![CDATA[Job Scheduling]]></category>
		<category><![CDATA[journalctl]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Reliability Engineering]]></category>
		<category><![CDATA[Shell Scripting]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[systemd Timers]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=254</guid>

					<description><![CDATA[<p>Both cron and systemd timers can run a command at 2 a.m. The difference is what happens the night it doesn't run, and how long it takes anyone to notice. A comparison by failure family, with a decision rule, the same job written both ways, and how to debug each one when it goes quiet.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/cron-vs-systemd-timers/">Cron vs systemd Timers: The Failure Modes That Decide It</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket usually arrives from the wrong direction. Nobody reports &#8220;the scheduler is broken.&#8221; Somebody asks for a restore, and the newest dump in the backup directory is three weeks old.</p>



<p class="wp-block-paragraph">The crontab entry is still there. The script still works when you run it by hand. Nothing in the monitoring went red, because nothing was watching whether the job ran at all. It was watching whether the server was up.</p>



<p class="wp-block-paragraph">That is the real question behind cron vs systemd timers, and it has almost nothing to do with syntax. Both tools can run a command at 2 a.m. The difference is what happens on the night the command doesn&#8217;t run, and how long it takes anyone to find out.</p>



<p class="wp-block-paragraph">This post compares the two by failure family rather than by feature list: missed runs, overlapping runs, environment surprises, output that goes nowhere, and jobs that eat the machine. Then a decision procedure, the same job written both ways, and how to debug each one when it goes quiet.</p>



<h2 class="wp-block-heading">The failure that actually decides it</h2>



<p class="wp-block-paragraph">Scheduled jobs fail silently far more often than they fail loudly. A job that crashes with a stack trace gets fixed the same week. A job that stopped firing six weeks ago gets found during an incident.</p>



<p class="wp-block-paragraph">Cron&#8217;s default answer to &#8220;what happened?&#8221; is to mail the job&#8217;s output to the owning user. On a server with no local mail transfer agent configured, which is most servers built in the last decade, that output is generated and then discarded. The daemon logs that it <em>started</em> the command. It does not log what the command printed, and in most implementations it does not log the exit status either.</p>



<p class="wp-block-paragraph">So you get a log line saying cron ran your backup, every night, forever, including on the nights the backup wrote nothing. That line is the trap. It looks like evidence.</p>



<p class="wp-block-paragraph">systemd timers hand this to journald instead. Standard output and standard error from the triggered service land in the journal, tagged with the unit name, alongside the exit status and the runtime. That is the single biggest practical difference between the two, and it is the reason most of my new scheduled work goes into timers.</p>



<p class="wp-block-paragraph">Neither one alerts you. Both will happily not run for a month while you sleep well. Observability is not monitoring, and I&#8217;ll come back to that.</p>



<h2 class="wp-block-heading">Where cron still wins</h2>



<p class="wp-block-paragraph">Cron is one line in one file. There is no unit to reload, no second file to keep in sync, no install section to get wrong. <code>crontab -e</code>, five fields, a command, done. For a job that trims a log directory on a box you own, that economy is not nothing.</p>



<p class="wp-block-paragraph">It also runs where systemd does not. BSD hosts, Alpine images, containers, appliance firmware, embedded boxes, older enterprise systems still on a legacy init. If your configuration management has to cover a fleet that is not uniformly systemd, cron is the common denominator and it is not close.</p>



<p class="wp-block-paragraph">Per-user scheduling is genuinely simpler too. A user runs <code>crontab -e</code> and owns their own jobs, with no root involvement and no unit files in <code>/etc</code>. On shared hosting and on control panels like DirectAdmin or cPanel, this is the only model available, and the panel writes the crontab for you.</p>



<h3 class="wp-block-heading">Where cron loses</h3>



<ul class="wp-block-list">

<li>No dependency ordering. Cron cannot wait for the network to be up or the database to be ready. It fires at the wall clock time and hopes.</li>


<li>No overlap protection. If the 2 a.m. run is still going at 3 a.m., the 3 a.m. run starts anyway, on top of it.</li>


<li>No resource containment. A runaway job competes with production for CPU, memory and disk with nothing standing between them.</li>


<li>Minute resolution, and no catch-up if the machine was off.</li>


<li>A syntax with sharp edges that fail quietly, which is the next section.</li>

</ul>



<h2 class="wp-block-heading">Where systemd timers win</h2>



<p class="wp-block-paragraph">A timer separates <em>when</em> from <em>what</em>. The <code>.timer</code> unit holds the schedule. The <code>.service</code> unit holds the job. That split feels like bureaucracy until the first time you need to run the job right now, out of band, without touching the schedule. Then it is just <code>systemctl start myjob.service</code>, and the run is logged exactly like a scheduled one.</p>



<p class="wp-block-paragraph">Everything systemd knows how to do to a service, it will do to a scheduled job. Ordering after other units. Restart-on-failure. Runtime caps. Memory and CPU limits through cgroups. A dedicated user. A private <code>/tmp</code>. A failure handler unit that fires when the job exits non-zero.</p>



<p class="wp-block-paragraph">You also get a schedule you can interrogate before it bites you. <code>systemctl list-timers</code> prints every timer on the box with its next and last elapse. <code>systemd-analyze calendar</code> takes an expression and tells you exactly when it would fire, which is the closest thing either scheduler has to a unit test.</p>



<h3 class="wp-block-heading">Where systemd timers lose</h3>



<ul class="wp-block-list">

<li>Two files, a <code>daemon-reload</code> and an <code>enable</code> for every job. That is real friction when you have forty small jobs.</li>


<li>The calendar syntax is more expressive than cron&#8217;s and harder to eyeball. You will reach for <code>systemd-analyze calendar</code> more often than you expect.</li>


<li>Portability is gone. These units run on systemd hosts and nowhere else.</li>


<li>User timers need lingering enabled, or they stop when the user logs out and never start at boot. This surprises people once, memorably.</li>


<li>The journal is not a log file. If your log shipping only tails <code>/var/log</code>, timer output is invisible to it until you point the shipper at journald.</li>

</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Cron vs systemd timers, by failure family</h2>



<h3 class="wp-block-heading">The machine was off at 2 a.m.</h3>



<p class="wp-block-paragraph">Cron has no memory. If the box was down, rebooting, or suspended when the schedule came round, that run is gone. Nothing catches it up and nothing records that it was skipped.</p>



<p class="wp-block-paragraph">The traditional fix is anacron, which tracks the last successful run per job in a timestamp file and runs the job shortly after boot if the interval has lapsed. It is why the scripts in <code>/etc/cron.daily</code> still run on a laptop that is closed every night. The catch is that anacron works in whole days and only covers the <code>cron.daily</code>, <code>cron.weekly</code> and <code>cron.monthly</code> directories by default. It does not help your custom job at 02:00.</p>



<p class="wp-block-paragraph">systemd folds that behaviour into a single directive:</p>



<pre class="wp-block-code"><code>[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true</code></pre>



<p class="wp-block-paragraph">With <code>Persistent=true</code>, systemd writes a timestamp under <code>/var/lib/systemd/timers/</code> each time the timer fires. On boot it compares that stamp against the schedule, and if a run was missed it triggers immediately. This matters more than it sounds on any VPS you reboot for kernel updates, and a great deal on hosts at providers like Contabo or InterServer where you take maintenance windows on their timetable rather than yours.</p>



<p class="wp-block-paragraph">One caveat worth knowing: &#8220;immediately&#8221; means immediately. If you have twelve persistent timers and the box has been off for a week, all twelve fire within seconds of boot, on a machine that is still starting services. Pair <code>Persistent=true</code> with <code>RandomizedDelaySec=</code> so the catch-up is spread out.</p>



<h3 class="wp-block-heading">Two copies of the same job</h3>



<p class="wp-block-paragraph">This is the one that corrupts data. A sync job normally takes four minutes and runs every five. One night the remote end is slow, the run takes eleven minutes, and cron starts two more copies while the first is still writing.</p>



<p class="wp-block-paragraph">Cron will not stop this. You have to do it yourself, and the standard tool is <code>flock</code> from util-linux:</p>



<pre class="wp-block-code"><code>*/5 * * * * /usr/bin/flock -n /var/lock/sync.lock /usr/local/bin/sync.sh</code></pre>



<p class="wp-block-paragraph"><code>-n</code> means fail immediately rather than queue behind the running copy. Without it you build a backlog of processes all waiting for the same lock, which turns an overlap problem into a fork bomb with a slow fuse.</p>



<p class="wp-block-paragraph">systemd gets this for free from its own model. A unit is either active or it isn&#8217;t. If <code>sync.service</code> is still running when the timer elapses, there is no second copy to start, because starting an already-active unit is a no-op. You do not have to remember anything.</p>



<p class="wp-block-paragraph">One default is worth knowing. A calendar timer schedules its next elapse from the previous trigger time, so if a run overran, that next elapse is already in the past and the service fires again the moment it finishes. Newer systemd releases add a <code>DeferReactivation=</code> boolean that schedules from when the service went inactive instead, so an overrunning job waits for the next real slot. Check your distribution&#8217;s systemd version before relying on it.</p>



<h3 class="wp-block-heading">The environment your job actually gets</h3>



<p class="wp-block-paragraph">&#8220;It works when I run it manually&#8221; is the single most common scheduled-job bug, and both schedulers cause it, for the same reason: neither one gives your job an interactive login shell.</p>



<p class="wp-block-paragraph">Cron runs the command with a short, hardcoded environment. Your shell profile is not sourced. Your <code>PATH</code> is much shorter than the one you tested with, which is why a script that calls <code>docker</code>, <code>wp</code>, <code>aws</code> or a version-manager shim works from your terminal and fails from cron. The working directory is the user&#8217;s home, not wherever you happened to be standing.</p>



<p class="wp-block-paragraph">Cron adds one hazard that is entirely its own. Inside a crontab, the percent sign is a metacharacter: it becomes a newline, and everything after the first one is fed to the command as standard input. So this looks fine and is broken:</p>



<pre class="wp-block-code"><code># Truncated at the first %. The redirect never happens.
0 2 * * * /usr/bin/mysqldump app &gt; /backups/app-$(date +%F).sql

# Escaped. This is what you meant.
0 2 * * * /usr/bin/mysqldump app &gt; /backups/app-$(date +%F).sql</code></pre>



<p class="wp-block-paragraph">There is no warning. The line parses, cron runs the truncated fragment, and the log says the job started. Anything with a <code>date +</code> format string in a crontab deserves a second look.</p>



<p class="wp-block-paragraph">systemd services get a minimal environment too, so the class of bug is identical. What differs is that the fix is declarative and lives with the unit rather than being hidden inside a wrapper script:</p>



<pre class="wp-block-code"><code>[Service]
Type=oneshot
User=deploy
WorkingDirectory=/srv/app
Environment=PATH=/usr/local/bin:/usr/bin:/bin
EnvironmentFile=/etc/myapp/backup.env
ExecStart=/usr/local/bin/backup.sh</code></pre>



<p class="wp-block-paragraph">The honest summary: absolute paths for every binary solve most of this on either scheduler. The rest is solved by never putting logic in the schedule. Put it in a script and schedule the script.</p>



<h3 class="wp-block-heading">Everything firing at once</h3>



<p class="wp-block-paragraph">Write <code>0 0 * * *</code> on fifty hosts and fifty hosts hit your backup target, your package mirror or your API at exactly midnight. The classic cron answer is to hash the hostname into a minute offset in configuration management, which works and which every team reinvents.</p>



<p class="wp-block-paragraph">systemd has two directives here and they pull in opposite directions, which is the part people get wrong:</p>



<ul class="wp-block-list">

<li><code>RandomizedDelaySec=</code> spreads timers <em>apart</em>. It adds a delay between zero and the value you set, so identical schedules on many hosts land at different moments.</li>


<li><code>AccuracySec=</code> pulls timers <em>together</em>. It lets systemd shift the firing time within a window to coalesce wakeups and save power. It defaults to one minute, which is why a timer set to 02:00:00 often fires a few seconds late.</li>

</ul>



<p class="wp-block-paragraph">If you want a timer to fire close to the stated second, lower <code>AccuracySec=</code>. If you want to stagger a fleet, raise <code>RandomizedDelaySec=</code>. Setting only <code>AccuracySec=</code> to a large value and expecting it to spread load is the mistake, because coalescing is the opposite of spreading.</p>



<h3 class="wp-block-heading">A job that eats the box</h3>



<p class="wp-block-paragraph">Cron has no answer here beyond <code>nice</code> and <code>ionice</code> in front of the command, and a script that hangs forever will hang forever. A systemd service is a cgroup, so the limits are declarative and enforced:</p>



<pre class="wp-block-code"><code>[Service]
Type=oneshot
MemoryMax=512M
CPUQuota=40%
IOWeight=20
TimeoutStartSec=30min
OnFailure=notify-failure@%n.service
ExecStart=/usr/local/bin/reindex.sh</code></pre>



<p class="wp-block-paragraph">The two directives that earn their keep on a shared box are <code>TimeoutStartSec=</code>, which kills a <code>oneshot</code> job that has hung, and <code>OnFailure=</code>, which starts another unit when this one exits non-zero. That second one is how you get a scheduled job to page you without wiring an alerting call into every script.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The same job written both ways</h2>



<p class="wp-block-paragraph">A nightly database dump that must not overlap, must not run before the database is up, and must catch up if the host was rebooted. In cron:</p>



<pre class="wp-block-code"><code>0 2 * * * /usr/bin/flock -n /var/lock/db-backup.lock /usr/local/bin/db-backup.sh &gt;&gt; /var/log/db-backup.log 2&gt;&amp;1</code></pre>



<p class="wp-block-paragraph">That handles overlap and captures output. It does not handle &#8220;the database wasn&#8217;t ready&#8221; or &#8220;the host was down&#8221;, and the log file is now your problem to rotate.</p>



<p class="wp-block-paragraph">The systemd version is two files. First the service, which is the job:</p>



<pre class="wp-block-code"><code># /etc/systemd/system/db-backup.service
[Unit]
Description=Nightly application database dump
After=network-online.target mariadb.service
Wants=network-online.target

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/db-backup.sh
TimeoutStartSec=45min</code></pre>



<p class="wp-block-paragraph">Then the timer, which is only the schedule:</p>



<pre class="wp-block-code"><code># /etc/systemd/system/db-backup.timer
[Unit]
Description=Run the nightly database dump

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=15min

[Install]
WantedBy=timers.target</code></pre>



<p class="wp-block-paragraph">Because the two units share a base name, you do not need a <code>Unit=</code> line. systemd pairs <code>db-backup.timer</code> with <code>db-backup.service</code> by convention. Add <code>Unit=</code> only when the names differ.</p>



<p class="wp-block-paragraph">Then the part people skip, which is verifying it before walking away:</p>



<pre class="wp-block-code"><code># Check the units parse before you enable anything
systemd-analyze verify /etc/systemd/system/db-backup.*

# Confirm the calendar expression means what you think
systemd-analyze calendar --iterations=5 "*-*-* 02:00:00"

systemctl daemon-reload
systemctl enable --now db-backup.timer

# Next and last elapse for this timer
systemctl list-timers db-backup.timer

# Run it now, out of band, without touching the schedule
systemctl start db-backup.service
journalctl -u db-backup.service -n 50 --no-pager</code></pre>



<p class="wp-block-paragraph">Note that you enable the <em>timer</em>, not the service. Enabling the service instead would try to run the job at every boot, which is a memorable way to learn the difference.</p>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">Work down this list and stop at the first line that applies. It resolves most cases in under a minute.</p>



<ol class="wp-block-list">

<li>The host might not be systemd, now or later. Use cron. Portability beats every other consideration on this list.</li>


<li>The job is inside a container. Use neither. Use your orchestrator&#8217;s scheduler, or a timer on the host that invokes the container.</li>


<li>A missed run matters, or the job must not overlap, or it needs to wait for another service. Use a timer. Every one of those is a directive rather than a wrapper script.</li>


<li>The job is heavy: a dump, a reindex, an import, anything that can pin a core or fill memory. Use a timer, for the cgroup limits and the runtime cap.</li>


<li>You need to see output after the fact without building a logging path first. Use a timer. The journal is already there.</li>


<li>You need sub-minute scheduling. Use a timer with a monotonic directive. Cron&#8217;s floor is one minute.</li>


<li>Otherwise, cron is fine, and one line beats two files.</li>

</ol>



<p class="wp-block-paragraph">The migration question answers itself from that. Do not convert forty working crontab lines in a weekend. Write new scheduled work as timers, convert the jobs that have actually burned you, and leave the rest. Mixing both on one host is normal and supported. Just never schedule the same job in both places, which is easier to do than it sounds when a configuration management run and a manual edit disagree.</p>



<h2 class="wp-block-heading">Diagnosing a job that never fired</h2>



<h3 class="wp-block-heading">Cron</h3>



<p class="wp-block-paragraph">Establish what actually happened before you theorise. The daemon logs every job it starts, so the log tells you which of three situations you are in: cron never tried, cron tried and the command failed, or cron never saw the entry at all.</p>



<pre class="wp-block-code"><code># Is the daemon running at all?
systemctl status crond      # RHEL, AlmaLinux, Rocky
systemctl status cron       # Debian, Ubuntu

# What did it start, and when?
journalctl -u crond --since "24 hours ago"
grep CRON /var/log/syslog | tail -20     # Debian family
tail -50 /var/log/cron                   # RHEL family

# Whose crontab are you actually looking at?
crontab -l -u deploy
ls -l /etc/cron.d/</code></pre>



<p class="wp-block-paragraph">Then work through the usual suspects, roughly in order of how often they turn out to be the cause:</p>



<ul class="wp-block-list">

<li><strong>Wrong crontab.</strong> The job is in root&#8217;s crontab and you are reading the deploy user&#8217;s, or it&#8217;s in <code>/etc/cron.d</code> where there is an extra user field before the command.</li>


<li><strong>PATH, or an unescaped percent sign.</strong> Dump what the job really sees with a temporary entry like <code>* * * * * env &gt; /tmp/cron-env.txt 2&gt;&amp;1</code>, then compare it against your shell.</li>


<li><strong>Permissions.</strong> The script is not executable, or the crontab owner cannot read it.</li>


<li><strong>The schedule is not what you think.</strong> When both day-of-month and day-of-week are restricted, cron runs the job when <em>either</em> matches, not both. <code>0 0 15 * 1</code> fires on the 15th and on every Monday.</li>


<li><strong>The command captured no output</strong>, so a real error was mailed into the void. Append <code>&gt;&gt; /var/log/myjob.log 2&gt;&amp;1</code> and wait for the next run.</li>

</ul>



<h3 class="wp-block-heading">systemd</h3>



<pre class="wp-block-code"><code># Is the timer loaded and scheduled? --all shows inactive ones too
systemctl list-timers --all

# Timer state, last trigger, next elapse
systemctl status db-backup.timer

# What the job printed, and how it exited
journalctl -u db-backup.service --since "7 days ago"

# Prove the schedule independently of the unit
systemd-analyze calendar --iterations=3 "Mon..Fri *-*-* 06:30:00"</code></pre>



<ul class="wp-block-list">

<li><strong>The timer is not enabled.</strong> Creating the files does nothing. It needs <code>daemon-reload</code> and <code>enable --now</code>.</li>


<li><strong>You enabled the service instead of the timer.</strong> Check with <code>systemctl is-enabled</code> on both.</li>


<li><strong>A user timer with no lingering.</strong> Under <code>systemctl --user</code>, the manager stops when the session ends unless you run <code>loginctl enable-linger</code> for that user.</li>


<li><strong>The calendar expression is valid but wrong.</strong> Valid syntax that never matches will not error. <code>systemd-analyze calendar</code> is how you find out.</li>


<li><strong>The service failed before your script ran.</strong> A missing <code>User=</code>, an <code>ExecStart</code> path that does not exist, a <code>WorkingDirectory</code> that is not there. The journal names the reason.</li>

</ul>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<p class="wp-block-paragraph"><strong>&#8220;Cron is deprecated.&#8221;</strong> It isn&#8217;t. It is packaged and supported on every mainstream distribution, and control panels, hosting providers and countless application installers write crontab entries as a matter of course. Nobody is removing it.</p>



<p class="wp-block-paragraph"><strong>&#8220;Timers are too complicated for small jobs.&#8221;</strong> Two files is more typing, not more complexity. The complexity that matters is the wrapper script you write to get locking, logging and a runtime cap out of cron, and how well the next person understands it. Compare like for like.</p>



<p class="wp-block-paragraph"><strong>&#8220;The journal means I have logging sorted.&#8221;</strong> The journal is a ring buffer with a size cap. If nobody set <code>SystemMaxUse=</code> deliberately and nobody ships journal entries anywhere else, your evidence has a retention window you never chose. Check it before you rely on it.</p>



<p class="wp-block-paragraph"><strong>&#8220;Switching to timers will stop jobs failing silently.&#8221;</strong> This is the one I&#8217;d push back on hardest. Timers make failures <em>visible</em>. They do not make them <em>noticed</em>. Neither scheduler will ever tell you that a job did not run, because absence is not an event. That needs a dead man&#8217;s switch: the job pings a heartbeat endpoint on success, and something external alerts when the ping stops. Healthchecks.io, Cronitor and Better Stack all sell exactly this, and Healthchecks is open source if you would rather self-host it next to your own Grafana. If you already run Prometheus, writing a completion timestamp through the node exporter&#8217;s textfile collector and alerting on its age gets you the same property without another vendor.</p>



<p class="wp-block-paragraph"><strong>&#8220;Just move everything to timers.&#8221;</strong> A mass migration of working jobs is a lot of change for no new capability, and every converted job is a chance to fat-finger a path. Convert on cause, not on principle.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Are systemd timers better than cron?</h3>



<p class="wp-block-paragraph">For production jobs on a systemd host, generally yes: logging, dependency ordering, overlap protection, catch-up and resource limits are all built in rather than bolted on. For a one-line job on a box you own, cron&#8217;s simplicity is a real advantage and the timer buys you nothing you will use.</p>



<h3 class="wp-block-heading">Is cron deprecated or being removed?</h3>



<p class="wp-block-paragraph">No. Cron implementations are still packaged and maintained across the major distributions. Some distributions do not install a cron daemon in minimal images by default, which is a packaging decision rather than a deprecation, so check with your package manager before assuming a fresh host has one.</p>



<h3 class="wp-block-heading">Can cron and systemd timers coexist on the same server?</h3>



<p class="wp-block-paragraph">Yes, and most real servers run both. The cron daemon itself is usually managed as a systemd service. The only rule is that a given job should be scheduled in exactly one place. Double-scheduling is what produces the &#8220;why did this run twice&#8221; ticket.</p>



<h3 class="wp-block-heading">How do I see when a systemd timer will next run?</h3>



<p class="wp-block-paragraph"><code>systemctl list-timers</code> shows next and last elapse for all active timers; add <code>--all</code> to include inactive ones. To test an expression without creating a unit, use <code>systemd-analyze calendar "Mon..Fri *-*-* 06:30:00"</code> with <code>--iterations=N</code> for more fire times.</p>



<h3 class="wp-block-heading">How do I stop a cron job from overlapping itself?</h3>



<p class="wp-block-paragraph">Wrap it in <code>flock -n</code> with a dedicated lock file. The <code>-n</code> flag makes the new run exit straight away instead of queuing. Do not use a PID file you wrote yourself; it will leak a stale lock the first time the job is killed, and then the job stops running forever.</p>



<h3 class="wp-block-heading">Do systemd timers work inside Docker containers?</h3>



<p class="wp-block-paragraph">Not usefully. A container normally runs a single process without systemd as PID 1, so there is no service manager to own a timer. Schedule from outside the container instead: a host timer that runs the container, or the scheduler your platform provides. Podman is the exception worth knowing, since it can generate systemd units for containers and the host&#8217;s systemd drives them.</p>



<h3 class="wp-block-heading">How do I convert a crontab line to a systemd timer?</h3>



<p class="wp-block-paragraph">Move the command into a <code>Type=oneshot</code> service with an absolute <code>ExecStart</code>, then translate the five cron fields into an <code>OnCalendar=</code> expression of the form <code>DayOfWeek Year-Month-Day Hour:Minute:Second</code>. Verify the expression with <code>systemd-analyze calendar</code>, enable the timer, then remove the crontab line only after you have seen a successful run in the journal.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Cron vs systemd timers is not a contest between an old tool and a new one. Cron is a scheduler. A systemd timer is a scheduler attached to a supervisor, and the supervisor is what you are actually buying: dependency ordering, overlap protection, catch-up, resource limits, and output that lands somewhere you can read it.</p>



<p class="wp-block-paragraph">Pick timers when a missed or duplicated run costs you something. Pick cron when it doesn&#8217;t. And whichever you pick, add a heartbeat, because the failure that hurts is not the job that crashes. It&#8217;s the job that stopped running while everything stayed green.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your scheduled jobs?</h2>



<p class="wp-block-paragraph">Scheduling is one of those areas where the problem is rarely the syntax. It&#8217;s the job nobody has checked since the person who wrote it left. This is the kind of work I take on:</p>



<ul class="wp-block-list">

<li>Auditing every crontab and timer on a host and telling you which jobs have quietly stopped running</li>


<li>Converting fragile cron entries to systemd timers with locking, runtime caps and failure handlers, without a big-bang migration</li>


<li>Fixing the classic &#8220;works by hand, fails on schedule&#8221; environment and PATH problems, including the percent-sign trap</li>


<li>Adding heartbeat monitoring and alerting so a missed run pages someone instead of sitting there</li>


<li>Backup and dump jobs that verify their own output rather than exiting zero on an empty file</li>


<li>Untangling duplicate schedules left behind by control panels, deploy scripts and configuration management</li>

</ul>



<p class="wp-block-paragraph">If something on your server isn&#8217;t running when it should, send me the crontab line or the unit file and the journal output around the time it should have fired. That&#8217;s usually enough to tell you what&#8217;s wrong before we talk about anything else.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/cron-vs-systemd-timers/">Cron vs systemd Timers: The Failure Modes That Decide It</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>journald vs rsyslog: Where Your Linux Logs Actually Live</title>
		<link>https://john-nessime.com/blog/devops/journald-vs-rsyslog-where-logs-live/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 08 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Disk Space]]></category>
		<category><![CDATA[journalctl]]></category>
		<category><![CDATA[journald]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[Logs]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[rsyslog]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Syslog]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=168</guid>

					<description><![CDATA[<p>On a stock Linux server, journald and rsyslog are both running and both storing the same events, with retention policies that don't know about each other. Here's where each log line actually lives, the three failure modes that bite in production, and how to decide which copy you keep.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/journald-vs-rsyslog-where-logs-live/">journald vs rsyslog: Where Your Linux Logs Actually Live</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The disk alert usually comes first. <code>/var</code> sitting at 94 percent on a box that runs three services and logs almost nothing interesting. You go digging and find the same night&#8217;s SSH failures in two places: compressed binary files under <code>/var/log/journal/</code>, and plain text in <code>/var/log/secure</code>. Same events. Two copies. Nobody decided that. It is just what the distro shipped.</p>



<p class="wp-block-paragraph">The worse version of this problem is the mirror image. You reboot a server to clear something, come back to read what led up to it, and the journal starts at the reboot. Everything before is gone. Not rotated, not compressed, not archived somewhere clever. Gone, because the journal was living in RAM and nobody ever told it otherwise.</p>



<p class="wp-block-paragraph">Both of those are the same misunderstanding wearing different clothes: not knowing where your logs actually live. This post walks through the <strong>journald vs rsyslog</strong> split on a modern Linux server, what each one really stores, the three failure modes that bite in production, and how to decide which of the two is the copy you keep.</p>



<h2 class="wp-block-heading">What actually happens when a process logs a line</h2>



<p class="wp-block-paragraph">On any systemd-based distribution, <code>systemd-journald</code> is the first stop for almost everything. It is not one option among several. It sits underneath.</p>



<ul class="wp-block-list">
<li>Anything a unit writes to stdout or stderr, because systemd wires those to the journal by default.</li>

<li>Kernel messages, read from the kernel ring buffer.</li>

<li>Classic <code>syslog(3)</code> calls from libc, which land on the <code>/dev/log</code> socket that journald owns.</li>

<li>Native journal API calls from anything linked against libsystemd, which carry structured key-value fields instead of a flat string.</li>

<li>Audit records, if <code>Audit=yes</code> is in effect.</li>
</ul>



<p class="wp-block-paragraph">journald writes all of that into indexed binary journal files. Then, on most server distributions, rsyslog gets a <em>second copy</em> of the same events and writes them out as text into <code>/var/log/messages</code>, <code>/var/log/secure</code>, <code>/var/log/maillog</code> and friends.</p>



<p class="wp-block-paragraph">That handoff happens one of two ways, and which one your box uses matters more than most people expect. More on that below. First, the part that costs you an outage.</p>



<p class="wp-block-paragraph">Before changing anything, look at the effective journald configuration rather than the file you think is authoritative. Distributions ship drop-ins under <code>/usr/lib/systemd/journald.conf.d/</code> that quietly override upstream defaults:</p>



<pre class="wp-block-code"><code># Print the merged configuration, including every drop-in, in load order
systemd-analyze cat-config systemd/journald.conf</code></pre>



<p class="wp-block-paragraph">This is the single most useful command in this whole post. Half the arguments about &#8220;what the default is&#8221; evaporate once you run it, because upstream systemd and your distribution frequently disagree about <code>ForwardToSyslog</code>.</p>



<h2 class="wp-block-heading">Failure one: the journal that was never on disk</h2>



<p class="wp-block-paragraph">journald&#8217;s <code>Storage=</code> setting defaults to <code>auto</code>. That word does more work than it looks like. Under <code>auto</code>, journald writes to <code>/var/log/journal/</code> <em>only if that directory already exists</em>. If it does not, journald falls back to <code>/run/log/journal/</code>, which is tmpfs. Memory. Wiped on reboot.</p>



<p class="wp-block-paragraph">Nothing warns you. <code>journalctl</code> works fine, colours are pretty, filters work. You just silently have no history.</p>



<p class="wp-block-paragraph">The quickest test is boot history. If the journal is persistent you see multiple boots; if it is volatile you see exactly one:</p>



<pre class="wp-block-code"><code>journalctl --list-boots</code></pre>



<p class="wp-block-paragraph">To make it persistent, create the directory, let systemd-tmpfiles apply the correct ownership and ACLs, then restart the daemon and flush anything still held in RAM:</p>



<pre class="wp-block-code"><code>sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush
journalctl --list-boots</code></pre>



<p class="wp-block-paragraph">The <code>systemd-tmpfiles</code> step is the one people skip. The journal directory needs specific group ownership and ACLs so that unprivileged users can read their own entries and members of the <code>systemd-journal</code> group can read everything. Creating the directory with a bare <code>mkdir</code> and walking away usually works, but it is the kind of &#8220;usually&#8221; that produces a confusing permissions ticket six months later.</p>



<p class="wp-block-paragraph">If you want the behaviour to be explicit rather than inferred from a directory&#8217;s existence, set it in <code>/etc/systemd/journald.conf.d/00-storage.conf</code>:</p>



<pre class="wp-block-code"><code>[Journal]
Storage=persistent</code></pre>



<p class="wp-block-paragraph">A drop-in file is better than editing the main config, because package upgrades will not fight you over it.</p>



<h2 class="wp-block-heading">Failure two: paying for the same log line twice</h2>



<p class="wp-block-paragraph">Once the journal is persistent, you are storing everything twice on a default server build: once as compressed binary in the journal, once as text under <code>/var/log/</code>. Two stores, two completely separate retention policies, neither of which knows the other exists.</p>



<p class="wp-block-paragraph">The journal is capped by <code>SystemMaxUse=</code> and <code>SystemKeepFree=</code>. Left unset, they default to 10 percent and 15 percent of the filesystem respectively, and journald honours whichever is stricter. Recent systemd also caps those calculated defaults so they do not grow without bound on a large disk, which is worth knowing if you have ever wondered why a huge volume did not produce a proportionally huge journal.</p>



<p class="wp-block-paragraph">The text copy is capped by <code>logrotate</code>, configured somewhere completely different, usually <code>/etc/logrotate.d/rsyslog</code>, on a weekly or daily schedule with its own <code>rotate</code> count.</p>



<p class="wp-block-paragraph">Measure both before you tune either:</p>



<pre class="wp-block-code"><code># What the journal is actually consuming on disk
journalctl --disk-usage

# What everything under /var/log costs, sorted, biggest last
sudo du -sh /var/log/* | sort -h</code></pre>



<p class="wp-block-paragraph">On a small VPS this stops being academic quickly. A modest instance from a provider like InterServer, Hetzner or DigitalOcean often has <code>/var</code> sharing a single root volume of 20 to 40 GB. A journal allowed to take 10 percent of that, plus four weeks of rotated text logs, plus a database, plus a container image cache, is how you end up paged at two in the morning for something that is not a real incident.</p>



<p class="wp-block-paragraph">If you decide the text files are the copy you keep, put the journal on a short leash:</p>



<pre class="wp-block-code"><code>[Journal]
# Hard ceiling on total journal size
SystemMaxUse=200M
# Never let the journal push free space below this
SystemKeepFree=1G
# Discard entries older than this regardless of size
MaxRetentionSec=2day</code></pre>



<p class="wp-block-paragraph">Those three do different jobs and you generally want at least two of them. <code>SystemMaxUse</code> bounds the journal itself. <code>SystemKeepFree</code> protects the rest of the filesystem from the journal. <code>MaxRetentionSec</code> bounds it in time, which is what auditors and data-retention policies actually care about.</p>



<p class="wp-block-paragraph">To reclaim space immediately without waiting for the next rotation:</p>



<pre class="wp-block-code"><code># Trim archived journal files until the total falls below 500M
sudo journalctl --vacuum-size=500M

# Or trim by age instead
sudo journalctl --vacuum-time=7d</code></pre>



<p class="wp-block-paragraph">One caveat that surprises people: vacuuming only removes <em>archived</em> journal files. The currently active file is never deleted. If almost all your usage is in one large active file, run <code>journalctl --rotate</code> first, then vacuum.</p>



<h2 class="wp-block-heading">Failure three: two rate limiters, both silent</h2>



<p class="wp-block-paragraph">This is the one that is invisible until it matters, and it is the real reason to understand the journald vs rsyslog relationship rather than treating them as interchangeable.</p>



<p class="wp-block-paragraph">There are <em>two independent rate limiters</em> in the default pipeline, and both drop messages quietly.</p>



<h3 class="wp-block-heading">journald&#8217;s limiter</h3>



<p class="wp-block-paragraph">journald applies <code>RateLimitIntervalSec=</code> and <code>RateLimitBurst=</code> per service, defaulting to 30 seconds and 10000 messages. Exceed the burst inside the interval and the rest of that service&#8217;s messages in that window are discarded. Not queued. Discarded.</p>



<p class="wp-block-paragraph">The one piece of good news is that journald tells you, in the journal itself. Grep for it:</p>



<pre class="wp-block-code"><code>journalctl --grep="Suppressed" -n 50 --no-pager</code></pre>



<p class="wp-block-paragraph">A chatty web server or a container using the journald log driver will hit this. Rather than disabling rate limiting globally, override it for the one unit that needs it, using a drop-in:</p>



<pre class="wp-block-code"><code># /etc/systemd/system/nginx.service.d/logging.conf
[Service]
LogRateLimitIntervalSec=30s
LogRateLimitBurst=50000</code></pre>



<p class="wp-block-paragraph">Per-unit values override <code>journald.conf</code> for that unit only, which is exactly what you want. Turning the global limiter off means one runaway process can fill your disk and take the box down. Turning it up for the single service you know is noisy is a bounded decision.</p>



<h3 class="wp-block-heading">rsyslog&#8217;s limiter</h3>



<p class="wp-block-paragraph">If rsyslog is reading via the <code>imjournal</code> module, it applies <em>its own</em> rate limit on top, defaulting to 20000 messages per 600 seconds. A message can therefore survive journald&#8217;s limiter, land in the journal, and still never reach <code>/var/log/messages</code> or your remote log host.</p>



<p class="wp-block-paragraph">That is the nasty case, because your local investigation finds the events but your centralised alerting never fired. Two people look at two sources and reach opposite conclusions about whether something happened.</p>



<p class="wp-block-paragraph">rsyslog exposes counters for this through its statistics module, including how many messages it read from the journal, how many it submitted onward, and how many it discarded to rate limiting. Enable <code>impstat</code> if you run anything log-volume-sensitive; the discard counter is the number that tells you whether your pipeline is lying to you.</p>



<h2 class="wp-block-heading">How rsyslog gets its copy: imjournal or imuxsock</h2>



<p class="wp-block-paragraph">There are two mechanisms, they behave differently under load, and running both at once is a classic source of duplicate lines.</p>



<h3 class="wp-block-heading">imuxsock: journald pushes a copy to a socket</h3>



<p class="wp-block-paragraph">With <code>ForwardToSyslog=yes</code> in journald, journald writes a classic syslog-formatted copy of each message to a dedicated socket, and rsyslog reads it with <code>imuxsock</code>.</p>



<ul class="wp-block-list">
<li><strong>Where it wins:</strong> simple, no state file, no second rate limiter, and no possibility of rsyslog reading back its own output and looping.</li>

<li><strong>Where it does not:</strong> you get the flat syslog view only. Structured journal fields such as the originating unit are not carried across.</li>
</ul>



<h3 class="wp-block-heading">imjournal: rsyslog pulls from the journal</h3>



<p class="wp-block-paragraph"><code>imjournal</code> reads journal files directly and keeps a state file so it can resume where it left off after a restart.</p>



<ul class="wp-block-list">
<li><strong>Where it wins:</strong> structured fields survive the trip. If you want to route or template on the systemd unit name rather than parsing it out of a message string, this is the only option of the two.</li>

<li><strong>Where it does not:</strong> extra rate limiter to reason about, a state file that can drift, and a documented risk that a corrupted journal database causes rsyslog to re-read the same entries in a loop. rsyslog&#8217;s own documentation recommends using <code>imuxsock</code> instead unless you specifically need the structured data.</li>
</ul>



<p class="wp-block-paragraph">A representative <code>imjournal</code> load line, with the rate limit stated explicitly rather than left implicit:</p>



<pre class="wp-block-code"><code>module(load="imjournal"
       StateFile="imjournal.state"
       Ratelimit.Interval="600"
       Ratelimit.Burst="20000")</code></pre>



<p class="wp-block-paragraph">Writing the defaults out by hand is worth the two extra lines. The next person to debug missing messages will see the limiter exists instead of assuming there isn&#8217;t one.</p>



<p class="wp-block-paragraph">To find out which mechanism your box is using right now:</p>



<pre class="wp-block-code"><code>grep -R "imjournal|imuxsock" /etc/rsyslog.conf /etc/rsyslog.d/ 2&gt;/dev/null
grep -R "ForwardToSyslog" /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ /usr/lib/systemd/journald.conf.d/ 2&gt;/dev/null</code></pre>



<p class="wp-block-paragraph">If the first command returns both modules and the second returns <code>ForwardToSyslog=yes</code>, you have a duplication problem. Pick one path and disable the other.</p>



<h2 class="wp-block-heading">Only one of the two leaves the box</h2>



<p class="wp-block-paragraph">This is the argument that settles most journald vs rsyslog debates in practice. journald is a local store. In a stock server install it has no mechanism for shipping logs to another host. There are separate systemd components for journal upload and reception, and third-party projects that read the journal and forward it, but none of that is in the default path.</p>



<p class="wp-block-paragraph">rsyslog forwards as a first-class feature, and it does the part that actually matters: it buffers when the destination is unreachable.</p>



<pre class="wp-block-code"><code>global(workDirectory="/var/spool/rsyslog")

action(type="omfwd"
       target="logs.example.net"
       port="514"
       protocol="tcp"
       action.resumeRetryCount="-1"
       queue.type="LinkedList"
       queue.filename="fwd_loghost"
       queue.maxDiskSpace="1g"
       queue.saveOnShutdown="on")</code></pre>



<p class="wp-block-paragraph">Line by line, because these are not decoration:</p>



<ul class="wp-block-list">
<li><code>workDirectory</code> is where spool files get written. Without it, the disk queue has nowhere to go.</li>

<li><code>protocol="tcp"</code> rather than UDP. UDP syslog silently drops under congestion, which defeats the point of forwarding at all.</li>

<li><code>queue.type="LinkedList"</code> makes the action asynchronous so a slow destination does not block local processing.</li>

<li><code>queue.filename</code> is what actually enables disk assistance. In-memory queue first, spilling to disk when it fills. The name must be unique across every action in the config.</li>

<li><code>queue.maxDiskSpace</code> caps that spool. Set it, or a long outage at the far end fills the disk you were trying to protect.</li>

<li><code>queue.saveOnShutdown="on"</code> persists whatever is still queued across a service restart.</li>
</ul>



<p class="wp-block-paragraph">One honest caveat on <code>action.resumeRetryCount="-1"</code>. Infinite retry is the right default for &#8220;never lose a log line&#8221;, but there are reported cases where a relay under sustained back-pressure stops accepting new inbound messages while retrying forever. If your host is a relay rather than a leaf, test that behaviour deliberately with the destination firewalled off before you trust it.</p>



<p class="wp-block-paragraph">rsyslog is not the only way off the box. Grafana Alloy into Loki, Vector, and Fluent Bit all read the journal directly and speak modern backends, and hosted platforms such as Better Stack, Papertrail or Datadog will take a plain syslog stream if you would rather not run the storage side. The mechanism differs; the decision does not. Something has to own the write path off the machine, and journald on its own is not it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">journald vs rsyslog: how I would decide</h2>



<p class="wp-block-paragraph">Stop asking which is better. Ask which one is the copy you are willing to be judged on, then make the other one cheap.</p>



<ol class="wp-block-list">
<li><strong>Do logs need to leave this machine?</strong> If yes, rsyslog or a dedicated shipper owns the outbound path. That is settled before anything else.</li>

<li><strong>Do you troubleshoot mostly by unit?</strong> If <code>journalctl -u something -b</code> is your muscle memory, journald is your primary read path and should get the retention budget.</li>

<li><strong>Do existing tools read text files?</strong> Fail2ban, logwatch, older SIEM collectors and a lot of home-grown scripts parse <code>/var/log/*</code>. Turning rsyslog off breaks them quietly.</li>

<li><strong>How much disk do you actually have?</strong> Under about 40 GB, keeping two full copies is an unforced error. Cap one hard.</li>

<li><strong>Is tamper-evidence a requirement?</strong> journald supports Forward Secure Sealing, which detects after-the-fact modification of journal files. Shipping to an append-only remote store is the more commonly accepted answer, and the two are not mutually exclusive.</li>
</ol>



<p class="wp-block-paragraph">The configuration I reach for first on a general-purpose server: journald persistent with a hard cap for a few days of fast local querying, rsyslog kept for the syslog files that other tools expect plus reliable forwarding off-box, <code>imuxsock</code> as the handoff unless something genuinely needs structured fields, and logrotate retention trimmed down because the remote copy is the one that matters for anything older than a week.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Journal only shows the current boot.</strong> Volatile storage. <code>journalctl --list-boots</code> confirms it, and the fix is the persistent-storage sequence above.</li>

<li><strong>Events in journalctl but not in /var/log/messages.</strong> Either rsyslog is not reading from the journal at all, or the <code>imjournal</code> rate limiter is discarding. Check which input module is loaded first.</li>

<li><strong>Every line appears twice in the text logs.</strong> Both <code>imjournal</code> and <code>imuxsock</code> are active while <code>ForwardToSyslog=yes</code>. Disable one path.</li>

<li><strong>Bursts of messages disappear during incidents.</strong> That is a rate limiter, and incidents are exactly when services get loud. Search for suppression notices and raise the limit for that specific unit.</li>

<li><strong>journalctl reports corruption or behaves oddly.</strong> Run <code>journalctl --verify</code>. If files are damaged, <code>journalctl --rotate</code> starts a fresh active file so new writes are clean, then vacuum the bad archives.</li>

<li><strong>&#8220;Journal has been rotated since unit was started.&#8221;</strong> Not an error. Rotation happened mid-session, so <code>journalctl -u</code> cannot map the full range. Re-run the query without the unit filter or widen the time range.</li>

<li><strong>Disk still full after vacuuming.</strong> Vacuum skips the active journal file. Rotate first, then vacuum again.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Assuming the journal is persistent because <code>journalctl</code> returns results. It always returns results. The question is how far back.</li>

<li>Disabling rsyslog on a box where fail2ban, logwatch or a scraper still reads <code>/var/log/secure</code>. Nothing errors. Detection just stops.</li>

<li>Setting <code>RateLimitBurst=0</code> globally to &#8220;stop losing logs&#8221;, then having a crash-looping service fill the disk in an afternoon.</li>

<li>Editing <code>/etc/systemd/journald.conf</code> directly and being surprised when a vendor drop-in overrides it. Use a drop-in of your own with a name that sorts later.</li>

<li>Forwarding over UDP because it is the one-line version. It drops silently under exactly the load that produced the logs you wanted.</li>

<li>Enabling a disk-assisted queue without <code>queue.maxDiskSpace</code>, turning a remote outage into a local disk-full outage.</li>

<li>Tuning journald limits on a machine whose real problem is that one application logs every health check at info level. Fix the source first.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Make storage explicit. <code>Storage=persistent</code> or <code>Storage=volatile</code>, never left to <code>auto</code> on a server you care about.</li>

<li>Always set <code>SystemKeepFree</code> alongside <code>SystemMaxUse</code>. The first bounds the journal, the second protects everything else on the volume from it.</li>

<li>Decide consciously which store is authoritative, and write it in the runbook. Two stores with no stated owner means neither gets maintained.</li>

<li>Use exactly one journal-to-rsyslog path. Both is duplication; neither is silent data loss.</li>

<li>Override rate limits per unit, not globally.</li>

<li>Monitor <code>journalctl --disk-usage</code> as a metric, not as something you check after the alert.</li>

<li>Ship off-box with TCP and a bounded disk-assisted queue, and test it with the destination firewalled off before you rely on it.</li>

<li>Keep the local copy short and the remote copy long. Local disk is the expensive place to store history.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I just disable rsyslog and use journald only?</h3>



<p class="wp-block-paragraph">On a self-contained box with no remote logging and nothing parsing text files, yes, and it saves real disk. Before you do it, grep your configuration management and cron jobs for <code>/var/log/</code> paths, and check whether fail2ban or any monitoring agent reads those files. That is where this bites people.</p>



<h3 class="wp-block-heading">Can I disable journald and use rsyslog only?</h3>



<p class="wp-block-paragraph">Not meaningfully. journald is how systemd captures unit stdout and stderr, so it stays in the path regardless. What you can do is set <code>Storage=volatile</code> with a small <code>RuntimeMaxUse</code> so it holds a short in-memory window and rsyslog owns everything durable.</p>



<h3 class="wp-block-heading">Where are journald logs stored?</h3>



<p class="wp-block-paragraph"><code>/var/log/journal/</code> when persistent, <code>/run/log/journal/</code> when volatile. They are indexed binary files, not text, so <code>grep</code> and <code>tail</code> do not work on them directly. Use <code>journalctl</code>, or <code>journalctl -o json</code> if you want to pipe structured output into something else.</p>



<h3 class="wp-block-heading">Why does journalctl show entries that never reached /var/log/messages?</h3>



<p class="wp-block-paragraph">Three usual causes: rsyslog is not reading the journal at all, the <code>imjournal</code> rate limiter dropped them, or a severity filter such as <code>MaxLevelSyslog</code> or an rsyslog rule excluded that facility or priority. Check in that order.</p>



<h3 class="wp-block-heading">Is the binary journal format a problem for log analysis?</h3>



<p class="wp-block-paragraph">Locally, no. Structured fields and fast filtering are genuinely better than parsing text with regex. It becomes a problem when you need to move logs somewhere else, because almost every collector expects text or JSON. That conversion step is precisely the role rsyslog or a modern shipper plays.</p>



<h3 class="wp-block-heading">How much disk should the journal be allowed to use?</h3>



<p class="wp-block-paragraph">Work backwards from how far back you actually query locally. If you rarely look past yesterday, a few hundred megabytes with <code>MaxRetentionSec</code> set to two or three days is plenty. Anything older belongs in a remote store where it is cheaper and survives the machine.</p>



<h3 class="wp-block-heading">Does journald compress logs automatically?</h3>



<p class="wp-block-paragraph">Yes. <code>Compress=</code> defaults to enabled and applies to entries above a size threshold, which is one reason the journal is often smaller than the equivalent text files despite carrying more metadata. It does not remove the need for a size cap.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The journald vs rsyslog question is not a contest. On a stock server both are running, both are storing the same events, and the defaults were chosen by your distribution rather than by you. The failure modes all come from that: a journal in RAM that vanishes on reboot, two uncoordinated retention policies eating the same disk, and two rate limiters dropping messages without anyone noticing.</p>



<p class="wp-block-paragraph">Run <code>systemd-analyze cat-config systemd/journald.conf</code> and <code>journalctl --list-boots</code> on your servers. Two commands, thirty seconds, and you will know whether the logs you would reach for during an incident are actually there.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help sorting out your logging pipeline?</h2>



<p class="wp-block-paragraph">Logging is one of those areas where the defaults work well enough to hide the problem until an incident makes it expensive. If any of the above sounded familiar, these are the things I take on:</p>



<ul class="wp-block-list">
<li>Auditing journald and rsyslog on existing servers and telling you exactly where each log line lives, how long it survives, and what is being silently dropped.</li>

<li>Fixing volatile journals, runaway <code>/var</code> growth and duplicate text-and-binary storage on small VPS instances.</li>

<li>Setting up reliable off-box forwarding with disk-assisted queues, TCP or RELP transport, and TLS where it is required.</li>

<li>Migrating from text-file scraping to a structured pipeline into Loki, Elasticsearch or a hosted log platform, without losing the alerts you already depend on.</li>

<li>Tuning rate limits and retention per service so noisy applications stop hiding the messages that matter.</li>

<li>Writing the runbook that says which store is authoritative, so the next engineer does not have to reverse-engineer it during an outage.</li>
</ul>



<p class="wp-block-paragraph">Send me your <code>journald.conf</code>, your <code>rsyslog.conf</code>, or the output of <code>journalctl --disk-usage</code> and I will tell you what I see before we talk about scope.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/journald-vs-rsyslog-where-logs-live/">journald vs rsyslog: Where Your Linux Logs Actually Live</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
