<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Log Retention | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/log-retention/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/log-retention/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 20 Oct 2026 06:00:04 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Log Retention | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/log-retention/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret</title>
		<link>https://john-nessime.com/blog/cloud-computing/audit-evidence-repository-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 20 Oct 2026 06:00:04 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Append-Only Storage]]></category>
		<category><![CDATA[Audit Evidence]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Backup]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Document Retention]]></category>
		<category><![CDATA[Immutable Storage]]></category>
		<category><![CDATA[Legal Hold]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Service Control Policies]]></category>
		<category><![CDATA[WORM Storage]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=910</guid>

					<description><![CDATA[<p>Enabling S3 Object Lock on a bucket does not protect the objects already in it. This is how to build an audit evidence repository on AWS that survives the question auditors actually ask, covering retention modes you cannot undo, delete markers that hide protected versions, chain of custody, and the storage bill compliance mode makes permanent.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/audit-evidence-repository-aws/">Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that ends a smooth audit call is never &#8220;do you have the logs&#8221;. It&#8217;s &#8220;show me why nobody could have edited them.&#8221;</p>



<p class="wp-block-paragraph">That is a different question, and most evidence buckets fail it quietly. Versioning is on, the bucket is encrypted, access logging is enabled, and there is still an admin role in the account that can call <code>DeleteObjectVersion</code> on every object in it. Nothing is wrong until someone asks you to prove a negative.</p>



<p class="wp-block-paragraph">An audit evidence repository on AWS is not a bucket with a good name. It is a write path, a retention decision you cannot take back, and a set of guardrails that survive the person who built them leaving the company. This post covers how to build one with S3 Object Lock and AWS Backup Vault Lock, and more usefully, the five ways I have seen the design go wrong in ways nobody notices until the evidence is needed or the bill arrives.</p>



<h2 class="wp-block-heading">What &#8220;immutable&#8221; actually buys you on AWS</h2>



<p class="wp-block-paragraph">Three separate mechanisms get called immutability, and they protect different things.</p>



<ul class="wp-block-list">
<li><strong>S3 Object Lock</strong> stops an object version from being overwritten or deleted for a defined period. It works only on versioned buckets, and once you enable it on a bucket you cannot disable it or suspend versioning again.</li>



<li><strong>AWS Backup Vault Lock</strong> does the same job for recovery points in a backup vault, with its own retention floor and ceiling enforced at the vault level.</li>



<li><strong>Integrity validation</strong>, such as CloudTrail&#8217;s digest files, proves a file has not changed since delivery. That is a separate property from being undeletable, and auditors increasingly ask for both.</li>
</ul>



<p class="wp-block-paragraph">Object Lock is the core of an audit evidence repository on AWS because evidence usually arrives as files: exported reports, signed approvals, config snapshots, log archives. Vault Lock matters when the evidence is a backup of a database or an EBS volume. Integrity validation is what turns &#8220;we kept it&#8221; into &#8220;we kept it and here is the hash chain&#8221;.</p>



<h2 class="wp-block-heading">Failure one: picking the retention mode you cannot walk back</h2>



<p class="wp-block-paragraph">Object Lock has two retention modes and the difference is not cosmetic. In governance mode, a caller who holds the <code>s3:BypassGovernanceRetention</code> permission can shorten or remove retention and delete the version. In compliance mode, <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html">nobody can</a>, including the account root user, and the retention period cannot be shortened or downgraded to governance.</p>



<p class="wp-block-paragraph">Read that second sentence again with a typo in mind. Set seven years where you meant seven days, in compliance mode, and the only exit AWS documents is closing the account. That is not a bug. It is the entire point of the mode. But it means the mode is a business decision wearing a technical costume.</p>



<p class="wp-block-paragraph">How I decide: compliance mode only where an external rule names a retention period and an external party will inspect it. Everything else starts in governance mode, with the bypass permission granted to nobody by default and denied outright in a service control policy. Governance mode with no principal holding the bypass right is functionally immutable for day-to-day purposes, and it leaves you a documented, auditable escape hatch that requires an explicit policy change to use.</p>



<p class="wp-block-paragraph">The bypass is not silent, either. Overriding governance retention requires the <code>x-amz-bypass-governance-retention:true</code> header on the request, which means the action shows up in CloudTrail as a distinctly shaped call. That is a good alarm to build.</p>



<pre class="wp-block-code"><code># Create the bucket with Object Lock on. Versioning is enabled automatically
# and cannot be suspended afterwards.
aws s3api create-bucket 
  --bucket my-evidence-archive 
  --object-lock-enabled-for-bucket 
  --region eu-west-1 
  --create-bucket-configuration LocationConstraint=eu-west-1

# Default retention for NEW objects written to the bucket.
aws s3api put-object-lock-configuration 
  --bucket my-evidence-archive 
  --object-lock-configuration 
  '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":365}}}'</code></pre>



<p class="wp-block-paragraph">Start there. Run it for a quarter. Watch what actually lands in the bucket and how big it gets. Then decide whether compliance mode is worth its permanence.</p>



<h2 class="wp-block-heading">Failure two: retention that was never applied to the objects that matter</h2>



<p class="wp-block-paragraph">This is the invisible one, and it is the reason to read the rest of this post.</p>



<p class="wp-block-paragraph">Object Lock is enabled at the bucket level, but retention lives on the individual object version. A default retention rule applies to objects written after the rule exists. It does not reach backwards. Enable Object Lock on an existing bucket holding three years of evidence and you get a bucket that reports Object Lock as enabled while every object already in it carries no retention at all.</p>



<p class="wp-block-paragraph">The bucket console looks correct. The compliance checklist gets ticked. The old evidence is still deletable by anyone with delete permissions.</p>



<p class="wp-block-paragraph">Two things to do about it. First, backfill retention on existing versions, either object by object with <code>put-object-retention</code> or, for anything above a few thousand keys, with S3 Batch Operations, which exists precisely for applying retention at scale. Second, verify per object rather than per bucket.</p>



<pre class="wp-block-code"><code># Check what protection a specific version actually carries.
# Look for ObjectLockMode, ObjectLockRetainUntilDate, ObjectLockLegalHoldStatus.
aws s3api head-object 
  --bucket my-evidence-archive 
  --key 2024/q3/access-review-export.csv 
  --version-id "SOME_VERSION_ID"</code></pre>



<p class="wp-block-paragraph">The stronger fix is to stop trusting the writer to set retention correctly and enforce a floor in the bucket policy. S3 exposes the <code>s3:object-lock-remaining-retention-days</code> condition key for exactly this, so you can reject writes that ask for too short a lock.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RejectShortRetention",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-evidence-archive/*",
      "Condition": {
        "NumericLessThan": {
          "s3:object-lock-remaining-retention-days": "365"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The write path now fails loudly when someone gets it wrong, which is what you want. A rejected upload gets noticed. A silently unprotected object does not.</p>



<h2 class="wp-block-heading">Failure three: the delete that looks like a delete</h2>



<p class="wp-block-paragraph">Object Lock protects object versions. It does not stop new versions being created, and it does not stop delete markers being placed on top of a protected object.</p>



<p class="wp-block-paragraph">So a user with ordinary delete permissions can issue a delete against a locked object, get a success response, and make it vanish from every normal listing. The protected version is still there. Nothing was destroyed. But an auditor browsing the bucket sees nothing, and the person who ran the command believes the file is gone.</p>



<p class="wp-block-paragraph">Three practical consequences:</p>



<ul class="wp-block-list">
<li>Deny <code>s3:DeleteObject</code> on the evidence prefix for everyone except a break-glass role. Immutability is not a substitute for permissions, it is a backstop under them.</li>



<li>Any retrieval tooling you hand to an auditor must list object <em>versions</em>, not objects. <code>list-object-versions</code>, not <code>list-objects-v2</code>.</li>



<li>Lifecycle rules interact with this in a way that surprises people. An expiration rule on a versioned bucket adds a delete marker rather than removing data, and a rule that tries to permanently expire a noncurrent version will not remove a version still under retention. Your cleanup policy and your retention policy are two different clocks, and the lock always wins.</li>
</ul>



<h2 class="wp-block-heading">Failure four: evidence that arrives already unprovable</h2>



<p class="wp-block-paragraph">Immutability starts when the object lands. Everything before that is chain of custody, and it is where most repositories are weakest.</p>



<p class="wp-block-paragraph">If the evidence is CloudTrail, use the integrity validation CloudTrail already ships. With log file validation enabled, CloudTrail writes an hourly digest file containing hashes of the log files delivered in that hour, signs each digest with a per-region private key, and puts the digests in a separate folder in the same bucket. You can then verify the chain later.</p>



<pre class="wp-block-code"><code># Turn on digest files for an existing trail.
aws cloudtrail update-trail 
  --name org-audit-trail 
  --enable-log-file-validation

# Verify a time range later. Files must still be in the location
# CloudTrail delivered them to, or validation cannot run.
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-audit-trail 
  --start-time 2020-01-01T00:00:00Z</code></pre>



<p class="wp-block-paragraph">Two details worth knowing before you design around it. Validation only covers log files referenced by digest files, so a gap in digest coverage is a gap in your proof. And the files must stay where CloudTrail put them, which means copying logs into a tidier prefix breaks the built-in verification and hands you the job of writing your own checker.</p>



<p class="wp-block-paragraph">For evidence that is not CloudTrail, the equivalent is doing the work yourself: hash the artifact at the point of generation, record the hash somewhere separate from the artifact, and write both. A manifest per collection run, checksummed and locked alongside the files, is unglamorous and holds up.</p>



<p class="wp-block-paragraph">One constraint that catches people mid-build: a bucket with Object Lock enabled cannot be used as the destination for S3 server access logs. If your design had one bucket receiving both the evidence and its own access logs, it needs splitting.</p>



<h2 class="wp-block-heading">Failure five: the bill you cannot switch off</h2>



<p class="wp-block-paragraph">Compliance mode plus a long retention period plus archival storage classes is where cost surprises live, because every lever you would normally pull to reduce spend has been deliberately disabled.</p>



<p class="wp-block-paragraph">The mechanics that drive it, none of which are secret but all of which get discovered late:</p>



<ul class="wp-block-list">
<li><strong>Minimum storage durations.</strong> S3 Glacier Flexible Retrieval carries a 90 day minimum and S3 Glacier Deep Archive carries 180 days. Transition earlier than that and you are billed for the remainder of the minimum anyway.</li>



<li><strong>Per-object overhead in Glacier classes.</strong> AWS documents 40 KB of metadata overhead per object for Glacier Flexible Retrieval and Deep Archive, of which 8 KB is billed at S3 Standard rates and 32 KB at the destination rate. For a repository full of 2 KB JSON approvals, the overhead is the bill.</li>



<li><strong>Billing starts when the rule is satisfied.</strong> Lifecycle transitions to Glacier classes happen asynchronously, but you are charged at the destination rate, and the minimum duration clock starts, from the date the rule matched rather than the date the bytes physically moved.</li>



<li><strong>You cannot delete your way out.</strong> Under compliance mode, oversized or mistakenly retained objects stay until their retain-until date passes, whatever the finance conversation says.</li>
</ul>



<p class="wp-block-paragraph">The design response is to aggregate before you lock. Batch small artifacts into per-day or per-collection archives, checksum the archive, and lock the archive rather than thousands of tiny files. It costs you a little retrieval convenience and saves you a per-object overhead multiplier you cannot renegotiate later.</p>



<p class="wp-block-paragraph">Also pick the retrieval tier against the real deadline. Deep Archive is the cheapest place to keep evidence and one of the slowest to get it back. If a regulator or a legal team may need files at short notice, that trade is a bad one regardless of the storage line item.</p>



<h2 class="wp-block-heading">Putting the repository together</h2>



<p class="wp-block-paragraph">With the failure families in view, the build order is fairly short.</p>



<ol class="wp-block-list">
<li><strong>Separate the account.</strong> Evidence lives in its own AWS account, not a prefix in the production log bucket. The blast radius of a compromised production admin should not include the record of what they did.</li>



<li><strong>Write the service control policies first.</strong> At the organisation level, deny <code>s3:BypassGovernanceRetention</code>, deny <code>s3:PutBucketPolicy</code> and <code>s3:DeleteBucketPolicy</code> on the evidence bucket, and deny disabling the trail. SCPs apply to the account root too, which is the point.</li>



<li><strong>Create the bucket with Object Lock and a default governance retention.</strong> Block public access, enforce TLS with an <code>aws:SecureTransport</code> condition, and encrypt with a customer managed KMS key whose policy is written for this bucket alone.</li>



<li><strong>Enforce the retention floor in the bucket policy</strong> using the remaining-retention-days condition, so a misconfigured collector fails rather than writes something unprotected.</li>



<li><strong>Replicate to a second account or region.</strong> Object Lock works with S3 Replication, and the destination bucket must also have Object Lock enabled. This is your answer to region-level loss and to an account-level mistake.</li>



<li><strong>Add Vault Lock for backup-shaped evidence.</strong> Snapshots and database backups belong in a locked backup vault rather than being exported to S3 by hand.</li>



<li><strong>Alarm on the write path.</strong> A day with no new evidence objects is the failure that never pages anyone. Alert on absence, not just on errors.</li>
</ol>



<p class="wp-block-paragraph">Vault Lock deserves its own note because its compliance mode has a grace period that Object Lock does not. You choose a cooling-off window, which AWS requires to be at least three days, during which the lock can still be removed. After that the vault and its lock are immutable and cannot be changed or deleted by anyone, including AWS.</p>



<pre class="wp-block-code"><code># Compliance mode: the presence of --changeable-for-days is what makes it
# compliance rather than governance. Minimum value is 3 (72 hours).
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name evidence-vault 
  --min-retention-days 365 
  --max-retention-days 2555 
  --changeable-for-days 3</code></pre>



<p class="wp-block-paragraph">Use that window. Run a backup job through the vault, confirm the retention arithmetic against what the policy document actually requires, and only then let the clock run out.</p>



<p class="wp-block-paragraph">One more design question worth asking out loud: does every copy need to be inside AWS? A second, offline-ish copy on independent infrastructure answers a different threat than cross-region replication does, because it survives an account-level compromise. Providers like InterServer, Contabo and Hetzner sell cheap storage that can hold an encrypted archive, and several S3-compatible services implement Object Lock semantics of their own. Confirm the provider actually supports Object Lock rather than assuming an S3-compatible API implies it, because that varies.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">AccessDenied when deleting a version you are allowed to delete</h3>



<p class="wp-block-paragraph">Usually governance retention plus a missing bypass. The IAM permission <code>s3:BypassGovernanceRetention</code> and the explicit bypass on the request are both required. If the object is in compliance mode, there is no combination of permissions that will work, and the error is telling you the truth.</p>



<pre class="wp-block-code"><code># Only valid for governance mode, and only with the matching IAM permission.
aws s3api delete-object 
  --bucket my-evidence-archive 
  --key path/to/object 
  --version-id "SOME_VERSION_ID" 
  --bypass-governance-retention</code></pre>



<h3 class="wp-block-heading">Replication fails on an Object Lock bucket</h3>



<p class="wp-block-paragraph">The destination needs Object Lock enabled too. Replicated objects take the lock configuration of the source, so a destination without Object Lock cannot accept them. Check the destination bucket first, then the replication role&#8217;s permissions.</p>



<h3 class="wp-block-heading">Lifecycle rule appears to do nothing</h3>



<p class="wp-block-paragraph">Two likely causes. The versions it targets are still under retention, so expiration cannot remove them. Or the rule targets current versions on a versioned bucket, in which case it is adding delete markers and the data is still there as noncurrent versions, still being billed.</p>



<h3 class="wp-block-heading">validate-logs reports invalid or missing files</h3>



<p class="wp-block-paragraph">Confirm nothing moved the objects. Validation requires the log and digest files to be in the location CloudTrail delivered them to, and the command runs per region. A copy job that reorganised the prefix layout is the usual culprit.</p>



<h3 class="wp-block-heading">Object Lock cannot be enabled on the bucket</h3>



<p class="wp-block-paragraph">Versioning has to be enabled first, and Object Lock cannot be turned off later. If the bucket is currently a server access log destination, that has to move before Object Lock will work for you at all.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Enabling Object Lock on an existing bucket and assuming the objects already in it are now protected. They are not, until retention is applied to each version.</li>



<li>Choosing compliance mode on day one to look thorough, then discovering the retention period was wrong.</li>



<li>Treating Object Lock as an access control. It is a floor under your permissions, not a replacement for them.</li>



<li>Leaving the bypass permission attached to a general administrator role because it was needed once during testing.</li>



<li>Locking millions of tiny objects into Glacier classes and paying per-object overhead on all of them.</li>



<li>Storing evidence in the same account as the systems it describes, which makes the evidence only as trustworthy as that account&#8217;s admins.</li>



<li>Never testing retrieval, so the first restore attempt happens under audit pressure with an unfamiliar Glacier restore workflow.</li>
</ul>



<h2 class="wp-block-heading">Best practices for an audit evidence repository on AWS</h2>



<ul class="wp-block-list">
<li>Governance mode by default, compliance mode only where an external requirement names the period. Write down which rule drove the choice, in the same repository.</li>



<li>Enforce the retention floor in the bucket policy so the write path fails loudly instead of writing unprotected objects.</li>



<li>Use legal hold, not extended retention, for investigations. Legal holds have no expiry and are removed explicitly, which is the right shape for something that ends on an unknown date.</li>



<li>Aggregate small artifacts into archives with a checksummed manifest before locking them.</li>



<li>Replicate to a second account with Object Lock enabled, and check replication metrics rather than assuming.</li>



<li>Rehearse retrieval quarterly, including the Glacier restore step, and keep the output as evidence of the rehearsal.</li>



<li>Monitor for absence of writes. Tools like Amazon CloudWatch, Grafana or Datadog are all fine here. The alert that matters is &#8220;no evidence arrived today&#8221;.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I enable S3 Object Lock on a bucket that already exists?</h3>



<p class="wp-block-paragraph">Yes. S3 supports enabling Object Lock on existing buckets, and versioning must be enabled. The important caveat is that enabling it does not retroactively protect the objects already stored. You apply retention to those separately, per object or with S3 Batch Operations.</p>



<h3 class="wp-block-heading">What is the difference between governance mode and compliance mode?</h3>



<p class="wp-block-paragraph">Governance mode can be overridden by a principal holding <code>s3:BypassGovernanceRetention</code> who sends the bypass header. Compliance mode cannot be overridden by anyone, including the root user, and the retention period cannot be shortened or the mode downgraded.</p>



<h3 class="wp-block-heading">Does S3 Object Lock stop someone deleting the file?</h3>



<p class="wp-block-paragraph">It stops the protected version being destroyed. It does not stop a delete marker being placed on top, which hides the object from ordinary listings while the version remains. Keep delete permissions tight and list versions when you audit.</p>



<h3 class="wp-block-heading">Do I need Object Lock if I already have versioning and MFA delete?</h3>



<p class="wp-block-paragraph">They solve different problems. Versioning preserves history but a sufficiently privileged caller can still remove versions. Object Lock removes that possibility for the duration of the retention period, which is the property an auditor is asking about.</p>



<h3 class="wp-block-heading">How do I prove the evidence has not been modified?</h3>



<p class="wp-block-paragraph">For CloudTrail, enable log file integrity validation and keep the signed digest files with the logs, then run <code>aws cloudtrail validate-logs</code> for the period in question. For other artifacts, hash at generation time, store the manifest separately, and lock both.</p>



<h3 class="wp-block-heading">Can a lifecycle rule delete objects that are still locked?</h3>



<p class="wp-block-paragraph">No. Retention wins over lifecycle expiration. Plan the two together, because a cleanup rule that quietly does nothing for years is a cost problem you will not see until you look at storage growth.</p>



<h3 class="wp-block-heading">Is Vault Lock compliance mode reversible?</h3>



<p class="wp-block-paragraph">Only during the cooling-off window you set at creation, which must be at least three days. After the lock date, the vault and its lock cannot be changed or deleted by anyone, including AWS.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An audit evidence repository on AWS fails in a specific, quiet way: the bucket says Object Lock is enabled, and the objects inside carry no retention. Everything else in this post is downstream of that gap. Check protection at the object version, not at the bucket, and enforce the floor in the write path so the system tells you when it is wrong.</p>



<p class="wp-block-paragraph">Then pick the mode like the one-way door it is. Governance mode with a locked-down bypass gets you almost all of the protection and keeps a documented way back. Compliance mode is worth its permanence when a real external rule demands it, and expensive theatre when it does not.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or reviewing one of these?</h2>



<p class="wp-block-paragraph">Immutable retention is easy to enable and easy to get subtly wrong, and the mistakes are the kind you find out about at the worst moment. This is work I take on directly:</p>



<ul class="wp-block-list">
<li>Designing an evidence account with Object Lock, KMS, service control policies and a locked-down write path</li>



<li>Auditing an existing bucket to find object versions that carry no retention despite Object Lock being enabled</li>



<li>Backfilling retention across existing objects with S3 Batch Operations without locking in the wrong period</li>



<li>Modelling the long-run cost of a retention decision before compliance mode makes it permanent</li>



<li>Setting up cross-account replication, AWS Backup Vault Lock and CloudTrail integrity validation as one coherent design</li>



<li>Writing the retrieval runbook and rehearsing it, so the first restore is not during an audit</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion on what you already have, send me the bucket policy, the Object Lock configuration and the output of a <code>head-object</code> call on one old evidence file. That is usually enough to tell you where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/">See how I work on projects</a></div>



<div class="wp-block-button is-style-outline is-style-outline--1"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/audit-evidence-repository-aws/">Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality</title>
		<link>https://john-nessime.com/blog/cloud-computing/cloudwatch-costs-logs-metrics-cardinality/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 23 Sep 2026 06:44:01 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Amazon Data Firehose]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cardinality]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost and Usage Report]]></category>
		<category><![CDATA[Cost Explorer]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Logs Insights]]></category>
		<category><![CDATA[Metric Filters]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Runaway Costs]]></category>
		<category><![CDATA[Showback]]></category>
		<category><![CDATA[VPC Flow Logs]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=785</guid>

					<description><![CDATA[<p>CloudWatch rarely fails loudly, it accumulates. This is a breakdown of the four independent meters behind the bill: ingestion, storage, Logs Insights scans and custom metric cardinality. Why the same log line gets billed three times, why log groups never expire by default, how a single dimension can multiply your metrics bill without your traffic changing, and how to attribute the spend to specific log groups before you start deleting things.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cloudwatch-costs-logs-metrics-cardinality/">The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in finance forwards the AWS invoice with a single line highlighted. AmazonCloudWatch, sitting uncomfortably close to your compute spend. Nobody launched anything. Nobody enabled a new feature. The number just kept climbing, in a slope gentle enough that nobody noticed until it was big enough to be somebody&#8217;s problem.</p>



<p class="wp-block-paragraph">That slope is the signature. CloudWatch rarely fails loudly, it accumulates: a log group created automatically by a service you enabled once, retention left at the default, a metric filter with a dimension nobody thought about, and a dashboard quietly calling the API every few seconds forever.</p>



<p class="wp-block-paragraph">This post breaks down where CloudWatch costs actually come from, why the same log line gets charged more than once, how cardinality multiplies your bill without your traffic changing, and how to attribute spend to specific log groups before you delete anything. It is organised by cost lever rather than by feature, because that is the order you work in when the invoice lands.</p>



<h2 class="wp-block-heading">The four meters behind your CloudWatch bill</h2>



<p class="wp-block-paragraph">CloudWatch is not one product with one price. It is a set of independent meters that happen to share a console. Working out which meter is running is most of the job, because the fix for each is completely different.</p>



<ul class="wp-block-list">
<li><strong>Ingestion.</strong> Per GB of log data accepted into a log group. Usually the largest component on log-heavy accounts.</li>

<li><strong>Storage.</strong> Per GB-month for what is retained afterwards. Cheap per unit, but it compounds because it never stops unless you tell it to.</li>

<li><strong>Analysis.</strong> Logs Insights bills on data scanned by a query, not on rows returned and not per query.</li>

<li><strong>Metrics and API calls.</strong> Custom metrics bill per unique time series per month. Separately, calls like GetMetricData bill per metric requested, which is how third-party dashboards end up on an AWS invoice.</li>
</ul>



<p class="wp-block-paragraph">The thing worth internalising: a single log line can be billed three times. Once when it arrives, again every month it sits in storage, and again every time a query scans the block it lives in. None of that is hidden, but nothing in the console surfaces it either.</p>



<h2 class="wp-block-heading">Ingestion is the meter that actually hurts</h2>



<h3 class="wp-block-heading">Uncompressed in, compressed out</h3>



<p class="wp-block-paragraph">Ingestion is metered on the uncompressed volume of events you send. Storage is billed on the compressed size CloudWatch keeps. Text logs compress well, so the stored figure can be a small fraction of what you were charged to get the data in.</p>



<p class="wp-block-paragraph">That asymmetry is why people look at a log group holding a few gigabytes, decide it is harmless, and miss that it cost several times that to arrive. The <code>storedBytes</code> value shown on a log group is the compressed number. It tells you about your storage line, not your ingestion line.</p>



<h3 class="wp-block-heading">The log groups you never created</h3>



<p class="wp-block-paragraph">Most runaway ingestion comes from things switched on with a checkbox. VPC Flow Logs pointed at CloudWatch. EKS control plane logging with every type enabled. Lambda functions emitting a full structured payload per invocation. CloudTrail delivering to a log group as well as S3. RDS general query logs left on after a debugging session.</p>



<p class="wp-block-paragraph">None of those feel like a decision at the time, and all of them scale with traffic rather than headcount. That is why the slope keeps rising while the team stays the same size.</p>



<p class="wp-block-paragraph">The most effective lever here is not a CloudWatch setting at all. It is cutting verbosity at the source, before the bytes leave your application. Dropping debug-level logging in production, sampling health check lines, trimming repeated stack frames. Filtering after ingestion does nothing for this meter, because you already paid.</p>



<h2 class="wp-block-heading">Retention: the one-line fix almost nobody applies</h2>



<p class="wp-block-paragraph">Log groups default to <strong>Never Expire</strong>. Not thirty days, not a year. Forever. Every log group created implicitly by another AWS service inherits that default, so an account accumulates hundreds of them without a single deliberate retention decision.</p>



<p class="wp-block-paragraph">Find the offenders first. This lists every log group with no retention policy, alongside its compressed size:</p>



<pre class="wp-block-code"><code>aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output table</code></pre>



<p class="wp-block-paragraph">The <code>?!retentionInDays</code> filter selects entries where the field is absent, which is how the API represents indefinite retention. There is no explicit &#8220;never&#8221; value to match on.</p>



<pre class="wp-block-code"><code>aws logs put-retention-policy 
  --log-group-name /aws/lambda/order-processor 
  --retention-in-days 30</code></pre>



<p class="wp-block-paragraph">Three things are worth knowing before running that across an account.</p>



<ol class="wp-block-list">
<li><strong>The value is an enum, not a free integer.</strong> Accepted values run 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731 and upward to ten years. Asking for 45 days is rejected.</li>

<li><strong>Deletion is not instant.</strong> Expired events are marked for deletion and typically removed within about three days. They stop contributing to your storage charge once marked, and are excluded from the reported <code>storedBytes</code>, so the billing effect lands before the data physically goes.</li>

<li><strong>Reverting means removing the policy, not setting zero.</strong> To go back to indefinite retention you call <code>delete-retention-policy</code> rather than passing a zero value.</li>
</ol>



<p class="wp-block-paragraph">Dry run before you bulk-apply. This prints what would change without touching anything:</p>



<pre class="wp-block-code"><code>aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].logGroupName' 
  --output text | tr 't' 'n' | while read -r lg; do
    echo "would set 30-day retention on: $lg"
  done</code></pre>



<p class="wp-block-paragraph">Retention is where compliance and cost pull against each other, and cost should lose. If an auditor needs seven years of logs, seven years is the answer. The move is to export them to S3 and apply lifecycle rules there, rather than paying CloudWatch rates for archival data you will query twice a decade.</p>



<h2 class="wp-block-heading">Cardinality: the multiplier nobody sees coming</h2>



<p class="wp-block-paragraph">This is the part that surprises people, because it breaks the intuition that cost tracks traffic. Cardinality can multiply your metrics bill by an order of magnitude while request volume stays flat.</p>



<p class="wp-block-paragraph">CloudWatch treats every unique combination of namespace, metric name and dimension values as a separate metric. Not a variant of one metric. A separate, separately billed metric. Publish <code>OrderLatency</code> with a <code>Region</code> dimension holding three values and you have three metrics. Add a <code>CustomerId</code> dimension and you have three times however many customers you have.</p>



<p class="wp-block-paragraph">The dangerous dimension values are the ones that look like identifiers: request IDs, trace IDs, pod names, session tokens, customer IDs, IP addresses, URL paths with IDs embedded in them. Each new value is a new billable series that persists and accrues.</p>



<h3 class="wp-block-heading">Metric filters have a guard rail, and it bites</h3>



<p class="wp-block-paragraph">Metrics extracted from logs by a metric filter bill as custom metrics. A filter can carry up to three dimensions, and AWS documents an explicit safety mechanism: CloudWatch Logs disables a metric filter that generates a thousand distinct dimension name/value pairs within a period.</p>



<p class="wp-block-paragraph">That guard rail is useful, but read what it means operationally. Hitting it does not just cost money, it silently stops the filter, so the metric you built an alarm on stops receiving data. If that alarm treats missing data as not breaching, it goes quiet rather than firing. You lose the alert and keep the bill.</p>



<p class="wp-block-paragraph">Audit what exists before adding more, and count what a namespace actually produces:</p>



<pre class="wp-block-code"><code>aws logs describe-metric-filters 
  --log-group-name /aws/lambda/order-processor

aws cloudwatch list-metrics 
  --namespace MyApp/Orders 
  --query 'length(Metrics)'</code></pre>



<p class="wp-block-paragraph">If that count is materially larger than the number of distinct things you consciously measure, you have found your multiplier.</p>



<h3 class="wp-block-heading">EMF does exactly what you tell it</h3>



<p class="wp-block-paragraph">Embedded Metric Format is a good design: you emit structured events, CloudWatch extracts metrics from declared fields, and you get high-cardinality context in the logs with low-cardinality metrics for alarming. That split is the point.</p>



<p class="wp-block-paragraph">The failure mode is promoting a high-cardinality field into the dimension set. AWS documents this plainly: declare something like a request ID as a dimension and EMF will, by design, create a custom metric for every unique combination. Keep identifiers in the log body as properties, and keep dimensions to values you would put on a dashboard axis.</p>



<h3 class="wp-block-heading">Anomaly detection alarms cost more than they look</h3>



<p class="wp-block-paragraph">An anomaly detection alarm does not evaluate one metric. It evaluates the source metric plus the upper and lower bounds of the predicted band, and those bounds are metrics too. That is roughly a threefold multiple against a standard threshold alarm, per alarm.</p>



<p class="wp-block-paragraph">It earns its keep on genuinely seasonal signals where a static threshold produces constant noise. Switching an entire alarm estate to it &#8220;to be safe&#8221; triples a line item for very little detection benefit.</p>



<h2 class="wp-block-heading">Query cost spikes exactly when you need it most</h2>



<p class="wp-block-paragraph">Logs Insights bills on the volume a query scans. Not rows returned. Not query count. Scanned bytes. A query returning twelve lines that scans a terabyte costs the same as one returning a million lines that scans a terabyte.</p>



<p class="wp-block-paragraph">During an incident, an engineer runs a dozen exploratory queries across a wildcard log group selection and a wide time window. The pricing model charges you hardest at the moment the tool is most valuable, which is an uncomfortable incentive. Two habits reduce it without reducing your ability to debug:</p>



<ul class="wp-block-list">
<li><strong>Narrow the time window first, then the filter.</strong> Scanning is bounded by the selected range across the selected groups. Starting at fifteen minutes and widening is far cheaper than starting at a week and narrowing.</li>

<li><strong>Select specific log groups, not patterns.</strong> Querying across every Lambda log group multiplies scanned bytes by the number of functions when you only care about one.</li>
</ul>



<p class="wp-block-paragraph">Watch for scheduled queries too. A dashboard widget or automation running an Insights query every few minutes against a large log group is a fixed monthly cost nobody remembers creating.</p>



<h2 class="wp-block-heading">Finding out where your CloudWatch costs come from</h2>



<p class="wp-block-paragraph">Attribute before you cut. Deleting log groups because they look big is how teams lose the one log they needed three weeks later.</p>



<ol class="wp-block-list">
<li><strong>Split the bill by meter.</strong> In Cost Explorer, filter service to CloudWatch and group by usage type. The region-prefixed usage types map to the meters directly: <code>DataProcessing-Bytes</code> is log ingestion, <code>TimedStorage-ByteHrs</code> is log storage, <code>DataScanned-Bytes</code> is Logs Insights, and <code>CW:Requests</code> and <code>CW:GMD-Metrics</code> cover API calls.</li>

<li><strong>Rank log groups by ingestion.</strong> The <code>IncomingBytes</code> metric in the <code>AWS/Logs</code> namespace, dimensioned by log group, is the per-group ingestion figure. Graph it with the Sum statistic. One documented gotcha: use an <em>absolute</em> range covering exactly thirty days, because a relative range or a different length produces a number that does not correspond to a billing month.</li>

<li><strong>Check stored bytes separately.</strong> A log group can be small in ingestion but large in storage because it has accumulated since the account was created. That is a retention problem, not a verbosity problem, and the fix differs.</li>

<li><strong>Count custom metrics per namespace.</strong> Compare the count against your mental model of what you measure. The gap is cardinality.</li>

<li><strong>Look inside the noisiest group.</strong> Once you know which log group dominates, find out which streams are responsible before changing any application code.</li>
</ol>



<p class="wp-block-paragraph">For that last step, a Logs Insights query over a short window is enough. Keep the range tight, because this query is itself billed on what it scans:</p>



<pre class="wp-block-code"><code>stats count(*) as events, sum(strlen(@message)) as approx_bytes by @logStream
| sort approx_bytes desc
| limit 20</code></pre>



<p class="wp-block-paragraph">That approximates byte volume per stream, which is usually enough to identify the chatty component. It is an approximation, not a billing figure, because it counts message characters rather than the metered event size.</p>



<p class="wp-block-paragraph">If you manage several accounts and want attribution done for you, cost platforms such as Vantage and CloudZero break CloudWatch spend down by usage type and resource without you building dashboards. Worth the spend if client reporting is part of the job. For one account and one engineer, Cost Explorer plus the steps above gets the same answer.</p>



<h2 class="wp-block-heading">Cheaper destinations, and what they cost you</h2>



<p class="wp-block-paragraph">Once you know your ingestion is dominated by logs you rarely read, there are three real options. None is free of trade-offs.</p>



<ul class="wp-block-list">
<li><strong>The Infrequent Access log class.</strong> Set at the log group level, it ingests at a materially lower per-GB rate in exchange for a reduced feature set. Historically that meant no metric filters, no alarming from logs and no Live Tail, though the capability list has expanded since launch. Check the current log classes documentation rather than assuming, and note that a log group you extract metrics from is not a candidate.</li>

<li><strong>Subscription filter to S3 via Amazon Data Firehose.</strong> Sends log data to object storage where lifecycle rules and Athena queries apply. You trade Insights and the console experience for much cheaper long-term retention. Good for audit and forensic logs, poor for anything you troubleshoot weekly.</li>

<li><strong>A separate log stack.</strong> Grafana Cloud, or self-hosted Loki and Prometheus on a VPS from a provider like Contabo or InterServer, moves cost from per-GB metering to a fixed monthly box you administer. The economics can be dramatically better at volume. The price is that you now own retention, backups, upgrades and the availability of your own observability during an outage, which is exactly when you least want to be debugging it.</li>
</ul>



<p class="wp-block-paragraph">The honest version: CloudWatch is expensive partly because it is operated for you and wired into everything. At a few dozen gigabytes a month, migrating away costs more engineering time than it saves. At terabytes, the arithmetic flips hard.</p>



<h2 class="wp-block-heading">Troubleshooting a bill that will not go down</h2>



<p class="wp-block-paragraph"><strong>You set retention everywhere and storage barely moved.</strong> Check whether storage was ever the dominant cost. If your usage type breakdown was mostly <code>DataProcessing-Bytes</code>, retention was never going to fix it.</p>



<p class="wp-block-paragraph"><strong>Custom metric count grows with no code changes.</strong> Something is emitting a dimension whose value set is open-ended. Look for deployment identifiers, container IDs, or a version string appended to a dimension.</p>



<p class="wp-block-paragraph"><strong>An alarm went quiet and the bill jumped the same week.</strong> Suspect a disabled metric filter. Check the filter still exists and is producing data, and verify the alarm&#8217;s missing data treatment. Alarms treating missing data as not breaching fail silently by design.</p>



<p class="wp-block-paragraph"><strong>API request charges you cannot place.</strong> External dashboards are the usual answer. A Grafana instance polling CloudWatch, or any tool calling GetMetricData on a short interval, bills per metric requested. Widening the refresh interval on dashboards nobody watches live is a quick win.</p>



<p class="wp-block-paragraph"><strong>Costs dropped, then came back.</strong> Look at your infrastructure as code. If Terraform or CloudFormation creates log groups without a retention argument, every apply restores the default. Fixing it in the console fixes it until the next deploy.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating storage as the problem because it is the number the console shows. Ingestion is usually larger and invisible in the log group list.</li>

<li>Adding a dimension because it would be useful to filter by, without checking how many distinct values it can take.</li>

<li>Filtering logs after they reach CloudWatch. The ingestion charge already applied.</li>

<li>Deleting log groups to cut cost without checking whether anything alarms on them.</li>

<li>Setting retention in the console while the IaC that owns the log group has none configured.</li>

<li>Enabling every EKS control plane log type when only the audit log is consumed.</li>
</ul>



<h2 class="wp-block-heading">Best practices for controlling CloudWatch costs</h2>



<ul class="wp-block-list">
<li><strong>Make retention a required argument.</strong> If log groups come from Terraform or CloudFormation, treat a missing retention setting as a policy violation and fail the plan. Checkov and similar scanners have rules for exactly this.</li>

<li><strong>Decide verbosity per environment, not per service.</strong> Debug in staging, info in production, with a documented way to raise it temporarily during an incident.</li>

<li><strong>Keep dimensions bounded, identifiers in the message body.</strong> If you cannot enumerate a dimension&#8217;s possible values on a whiteboard, it does not belong in the dimension set.</li>

<li><strong>Alarm on the bill itself.</strong> A billing alarm on estimated charges, or an anomaly monitor scoped to CloudWatch, turns a quarterly surprise into a same-week signal.</li>

<li><strong>Review the top five log groups by ingestion monthly.</strong> Five minutes with the <code>IncomingBytes</code> graph catches new noise before it compounds.</li>

<li><strong>Route archival logs out deliberately.</strong> Anything kept for compliance rather than operations belongs in S3 with lifecycle rules from the start.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Why is my CloudWatch bill so high when my traffic has not changed?</h3>



<p class="wp-block-paragraph">Two causes account for most cases. Storage accumulating because log groups default to indefinite retention, so the bill grows every month even at constant ingestion. Or metric cardinality, where a dimension with an open-ended value set keeps creating new billable series. Neither tracks request volume, which is why the bill and the traffic graph disagree.</p>



<h3 class="wp-block-heading">Does deleting a log group reduce my bill immediately?</h3>



<p class="wp-block-paragraph">It stops future storage charges for that data and nothing else. Ingestion already billed is spent, and deleting the group does not stop whatever was writing to it, which will simply recreate it. Turn off the source first, then delete.</p>



<h3 class="wp-block-heading">What counts as one custom metric for billing?</h3>



<p class="wp-block-paragraph">One unique combination of namespace, metric name and dimension values. The same metric name published with three different dimension value sets is three billable metrics. This is the single most important sentence in CloudWatch pricing and the easiest one to skim past.</p>



<h3 class="wp-block-heading">Should I switch log groups to the Infrequent Access class?</h3>



<p class="wp-block-paragraph">For logs you keep but rarely read, it is usually a straightforward saving on ingestion. For any group you extract metrics from, alarm on, or stream live, it is not a candidate. The feature gap between the classes has narrowed since launch, so check current documentation rather than ruling it out on old information.</p>



<h3 class="wp-block-heading">Is Logs Insights charged per query or per gigabyte?</h3>



<p class="wp-block-paragraph">Per gigabyte scanned. Rows returned are irrelevant. A tightly filtered query over a wide range across many log groups is expensive; a broad query over fifteen minutes on one log group is cheap. Narrow the range before you narrow the filter.</p>



<h3 class="wp-block-heading">Do VPC Flow Logs have to go to CloudWatch?</h3>



<p class="wp-block-paragraph">No. Flow logs can be delivered to S3 instead, avoiding CloudWatch ingestion and storage entirely, at the cost of losing Insights queries over them. If you analyse flow logs occasionally with Athena rather than continuously in the console, S3 is the better destination.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">CloudWatch costs are not one number with one cause. They are four meters running independently, and each has a different fix: cut verbosity at the source for ingestion, set retention for storage, narrow scope for queries, bound your dimensions for metrics. Applying the wrong fix to the wrong meter is why so many cost-reduction efforts produce a bill that barely moves.</p>



<p class="wp-block-paragraph">So attribute first. Split by usage type, rank log groups by <code>IncomingBytes</code>, count custom metrics per namespace. Ten minutes of that tells you which lever you are actually pulling, and stops you deleting the log group that turns out to be the only place a production alarm was getting its data.</p>



<h2 class="wp-block-heading">Need a hand with your CloudWatch spend?</h2>



<p class="wp-block-paragraph">This is well-defined work with a measurable outcome, which makes it a good fit for a short engagement. Things I can help with:</p>



<ul class="wp-block-list">
<li>Breaking your CloudWatch bill down by meter and by log group, so you know what you are paying for before anything changes</li>

<li>Auditing custom metric cardinality and identifying which dimensions are multiplying your series count</li>

<li>Building a retention policy that satisfies your compliance requirement without paying CloudWatch rates for archival data</li>

<li>Enforcing retention and log class in Terraform or CloudFormation so the fix survives the next deployment</li>

<li>Designing a subscription filter and S3 archive path for logs you need to keep but rarely query</li>

<li>Reviewing alarms and dashboards for anomaly detection and API polling costs that are not earning their keep</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me your Cost Explorer breakdown grouped by usage type, or the output of <code>describe-log-groups</code> for the account. That is usually enough to say where the money is going before we talk about anything else.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cloudwatch-costs-logs-metrics-cardinality/">The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Where PHI Escapes: Building a Secure Medical Lab Data Pipeline on AWS</title>
		<link>https://john-nessime.com/blog/cloud-computing/medical-lab-data-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 21 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon Comprehend Medical]]></category>
		<category><![CDATA[Amazon Macie]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS HealthLake]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[Business Associate Agreement]]></category>
		<category><![CDATA[CLIA]]></category>
		<category><![CDATA[Clinical Data]]></category>
		<category><![CDATA[Crypto Shredding]]></category>
		<category><![CDATA[Data Classification]]></category>
		<category><![CDATA[De-identification]]></category>
		<category><![CDATA[Dead Letter Queue]]></category>
		<category><![CDATA[ePHI]]></category>
		<category><![CDATA[FHIR]]></category>
		<category><![CDATA[Healthcare Cloud]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[HL7 v2]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Multi-Account Strategy]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Pseudonymization]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=455</guid>

					<description><![CDATA[<p>Encryption is the part everyone gets right. The PHI sitting in your logs, dead letter queues and object key names is the part that fails an audit. A boundary-by-boundary guide to building a medical lab data pipeline on AWS that survives review.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/medical-lab-data-pipeline-aws/">Where PHI Escapes: Building a Secure Medical Lab Data Pipeline on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that breaks most healthcare data designs is not technical. It is an auditor asking you to list every place protected health information is stored. Someone points at the encrypted S3 bucket, the encrypted RDS instance, the encrypted backups, and says &#8220;those three.&#8221; Then you grep CloudWatch Logs and find an accession number, a date of birth, and a patient surname sitting inside a Python traceback with no retention policy set on the log group.</p>



<p class="wp-block-paragraph">That is the failure mode that actually bites when you build a medical lab data pipeline on AWS. Not the bucket you thought about. The seven places PHI ended up because a payload got logged, an error got queued, or a filename contained a medical record number.</p>



<p class="wp-block-paragraph">This post walks through the design boundary by boundary: where lab data crosses from one trust zone into another, what leaks at each crossing, and which control closes it. It is aimed at engineers who have been handed an HL7 feed and a compliance checklist and told to make them meet. I am an engineer and not your compliance counsel, so treat the regulatory references here as pointers to read the primary text yourself.</p>



<h2 class="wp-block-heading">Why laboratory data is a harder pipeline than it looks</h2>



<p class="wp-block-paragraph">Lab data has three properties that most ETL work does not.</p>



<p class="wp-block-paragraph">First, results get corrected. A preliminary result goes out, the analyzer is recalibrated, and a corrected result follows hours or days later. Your pipeline is not append-only in the way a clickstream is. It has to carry amendment semantics all the way through, and an analytics table that silently keeps the first value is worse than no table at all.</p>



<p class="wp-block-paragraph">Second, the wire format is old and the transport is older. Most laboratory information systems still emit HL7 version 2 messages, pipe-delimited segments carried over the Minimum Lower Layer Protocol, which is framed bytes over a raw TCP socket. AWS documents MLLP as the common transport standard for HL7 v2 interoperability in its Healthcare Industry Lens. It has no built-in authentication and no built-in encryption. Whatever security it has, you bolt on around it.</p>



<p class="wp-block-paragraph">Third, the identifiers are load-bearing. Accession numbers, specimen IDs and medical record numbers are how the lab reconciles anything, so PHI is not a column you can drop. It is structural, and it will try to escape into every metadata surface you own.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Boundary one: getting messages out of the lab</h2>



<p class="wp-block-paragraph">The instrument or LIS sits on a network you probably do not control, run by a vendor who may not let you touch the configuration. That constrains the design more than anything on the AWS side.</p>



<p class="wp-block-paragraph">You get three realistic shapes, and the LIS vendor usually picks for you.</p>



<ul class="wp-block-list">
<li><strong>Live MLLP over a private link.</strong> An interface engine in the lab holds the TCP session and forwards over a Site-to-Site VPN or Direct Connect into a private subnet. Lowest latency, highest operational burden, because a dropped socket is a clinical incident and not a retry.</li>

<li><strong>Batch file drop over SFTP.</strong> The LIS writes flat files on a schedule and something picks them up. Far easier to reason about, far easier to audit, and acceptable whenever the downstream use is analytics rather than result delivery.</li>

<li><strong>An outbound API push.</strong> Newer systems will POST to an endpoint you expose. Best case, because you control authentication and you get a response code back, but rarest in practice.</li>
</ul>



<p class="wp-block-paragraph">Whichever shape you get, the same rule applies at this boundary: <strong>terminate the untrusted protocol as early as possible and convert to something you can authorize.</strong> A raw MLLP socket carries no identity. The moment those bytes land, wrap them in a message you can attribute, sign, and trace.</p>



<p class="wp-block-paragraph">Two things go wrong here. First, people treat a network tunnel as authentication. A VPN tells you the packets came from the lab&#8217;s network. It does not tell you which system sent them, or that the sender was authorized to send that patient&#8217;s results. Second, people assume consumer VPN products fill the gap. Tools like NordVPN or Surfshark are fine for your own admin laptop on untrusted wifi, but they carry no business associate agreement and they are not a substitute for AWS Site-to-Site VPN or AWS Client VPN on a clinical path.</p>



<p class="wp-block-paragraph">One piece of housekeeping that is cheap to do and expensive to skip: the business associate addendum. AWS makes this self-service through AWS Artifact, and it can be accepted across an entire organization. Only services on the current HIPAA Eligible Services Reference may hold PHI, and that list changes, so check it against the services you actually chose rather than your memory of it.</p>



<h2 class="wp-block-heading">Boundary two: the landing zone</h2>



<p class="wp-block-paragraph">Raw messages land in S3. This part is well documented, so I will focus on the parts people configure once and never verify.</p>



<h3 class="wp-block-heading">Make the bucket policy do the enforcing</h3>



<p class="wp-block-paragraph">Default bucket encryption means objects get encrypted. It does not mean a caller cannot write an object with different settings, and it does not stop a plaintext HTTP request. Two explicit denials cover the gap. The first protects the transport, the second protects the storage:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::lab-raw-inbound",
        "arn:aws:s3:::lab-raw-inbound/*"
      ],
      "Condition": {
        "Bool": { "aws:SecureTransport": "false" }
      }
    },
    {
      "Sid": "DenyWrongKey",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::lab-raw-inbound/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption-aws-kms-key-id":
            "arn:aws:kms:REGION:ACCOUNT:key/KEY-ID"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Use a customer managed KMS key, not the AWS managed one. The reason is not stronger cryptography, it is the key policy. A customer managed key gives you a second, independent authorization surface: even a principal with broad S3 permissions cannot read the object if the key policy does not grant them decrypt. That separation is what makes a blast radius argument credible to an assessor, and it is what makes crypto shredding possible later.</p>



<h3 class="wp-block-heading">Your object keys are metadata, and metadata leaks</h3>



<p class="wp-block-paragraph">This is the one I would put on a poster. Object contents are encrypted. Object <em>keys</em> are not. They show up in CloudTrail data events, S3 server access logs, S3 Inventory reports, bucket listings, Lambda event payloads, and every console screenshot anyone pastes into a ticket.</p>



<pre class="wp-block-code"><code># Leaks PHI into six places that are not the bucket
raw/2024/ORU_MRN4471902_SMITH_JOHN_A1C.hl7

# Same object, no identifiers in the key path
raw/dt=&lt;partition&gt;/src=lis01/msg=01HXYZ...ULID.hl7</code></pre>



<p class="wp-block-paragraph">Use an opaque, sortable identifier and keep the mapping from identifier to patient inside an encrypted store you actually control. Partition on ingest date and source system, never on anything derived from the patient. This one decision removes more PHI surface than any other single change in the pipeline.</p>



<h3 class="wp-block-heading">Immutability, and the trap inside it</h3>



<p class="wp-block-paragraph">Clinical laboratories in the United States operating under CLIA have record retention obligations set out in 42 CFR 493.1105. The floor for test reports is at least two years after the date of reporting, with pathology test reports at ten years, and several state requirements sit above the federal floor. Retention is therefore not something you invent; go and read the section, then check your state.</p>



<p class="wp-block-paragraph">S3 Object Lock gives you write-once-read-many enforcement, and it is the right tool. But understand the two modes before you commit, because one of them is genuinely irreversible.</p>



<ul class="wp-block-list">
<li><strong>Governance mode</strong> blocks deletion for most principals, but anyone holding <code>s3:BypassGovernanceRetention</code> can override it by sending the <code>x-amz-bypass-governance-retention:true</code> header. Note the S3 console includes that header by default.</li>

<li><strong>Compliance mode</strong> cannot be overridden by anyone, including the account root. Per AWS documentation, the only way to delete an object under compliance retention before its date expires is to delete the AWS account.</li>
</ul>



<p class="wp-block-paragraph">A typo in a retention date under compliance mode is permanent, and you pay to store the mistake for its full term. Object Lock also has to be enabled at bucket creation, alongside versioning. Start in governance mode, prove the retention values against real traffic, then move to compliance mode deliberately rather than as a default.</p>



<pre class="wp-block-code"><code># Indefinite hold for a specific object version, e.g. under litigation
aws s3api put-object-legal-hold 
  --bucket lab-raw-inbound 
  --key raw/dt=.../msg=01HXYZ.hl7 
  --legal-hold Status=ON</code></pre>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Boundary three: where PHI escapes into your observability</h2>



<p class="wp-block-paragraph">Here is the section that justifies the post. Everything above is on the compliance checklist. This is the part that is invisible until an audit or an incident makes it visible.</p>



<p class="wp-block-paragraph">Your operational plane is a PHI store you did not declare. Specifically:</p>



<ul class="wp-block-list">
<li><strong>Application logs.</strong> One <code>print(record)</code> left in a parser during debugging writes patient demographics into CloudWatch Logs, where the default is often no expiry at all.</li>

<li><strong>Exception traces.</strong> Unhandled parse errors frequently include the offending segment in the message. That segment is the PHI.</li>

<li><strong>Step Functions execution history.</strong> If you pass message bodies between states as payloads, that content is retained in the execution record and readable by anyone who can view executions.</li>

<li><strong>Dead letter queues.</strong> The whole point of a DLQ is to keep the failed message. A DLQ full of unparseable HL7 is a PHI archive with a different access policy than the bucket you designed so carefully.</li>

<li><strong>Third-party monitoring.</strong> Anything shipped to an external platform leaves your boundary. Grafana Cloud, Datadog and equivalents are perfectly reasonable choices for infrastructure telemetry, but a log line containing PHI going to a vendor without an executed BAA is a disclosure, not a metric.</li>

<li><strong>Notifications.</strong> An SNS topic that emails an on-call engineer &#8220;failed to process record for J. Smith, DOB &#8230;&#8221; has just sent PHI to a mail server nobody reviewed.</li>
</ul>



<h3 class="wp-block-heading">The pattern that fixes it: pass pointers, not payloads</h3>



<p class="wp-block-paragraph">Adopt a single rule and enforce it in review: <strong>PHI moves by reference between components, never by value.</strong></p>



<ol class="wp-block-list">
<li>The ingest function writes the raw message to the encrypted bucket and emits only an opaque object key plus a message ID.</li>

<li>Every downstream state, queue and event carries that pointer. Nothing carries the record body.</li>

<li>Error handling logs the pointer and a failure class, never the content that failed.</li>

<li>A DLQ therefore contains pointers to failures, and reprocessing means re-reading from the bucket under the same key policy as everything else.</li>
</ol>



<p class="wp-block-paragraph">The cost of this pattern is a slightly noisier debugging experience. You cannot read the failing record straight out of the queue. You have to go and fetch it, with credentials, and that fetch is logged. That inconvenience is the control working.</p>



<h3 class="wp-block-heading">Then close the log surface itself</h3>



<p class="wp-block-paragraph">Log groups created implicitly by Lambda and other services often retain data indefinitely unless you set a policy. Find the ones nobody configured:</p>



<pre class="wp-block-code"><code># List log groups with no retention configured
aws logs describe-log-groups 
  --query 'logGroups[?retentionInDays==null].logGroupName' 
  --output text

# Set an explicit retention on one of them
aws logs put-retention-policy 
  --log-group-name /aws/lambda/hl7-ingest 
  --retention-in-days 30</code></pre>



<p class="wp-block-paragraph">Run the first command as a scheduled check, not a one-off. New functions create new log groups, and the default comes back every time someone deploys.</p>



<p class="wp-block-paragraph">Amazon Macie is worth pointing at your buckets as a detective backstop. It will not stop a leak, but it will tell you when something started landing where it should not.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Boundary four: from identified data to analytics</h2>



<p class="wp-block-paragraph">At some point someone wants to query the data. This is the crossing where teams are most likely to convince themselves they have done something they have not.</p>



<p class="wp-block-paragraph">Amazon Comprehend Medical will detect PHI entities in unstructured clinical text through its <code>DetectPHI</code> operation, returning entity types such as name, address, identifier and date, each with a confidence score. It is a genuinely useful tool for narrative fields: pathology comments, specimen notes, microbiology free text.</p>



<pre class="wp-block-code"><code>aws comprehendmedical detect-phi 
  --text "Specimen received from the referring clinic on the stated date."</code></pre>



<p class="wp-block-paragraph">Now the part the marketing pages underplay, straight from the AWS developer guide: the entities Comprehend Medical detects <em>do not map one to one</em> to the identifier list specified by the Safe Harbor method, and AWS explicitly recommends additional human review or other methods to confirm accuracy for compliance use cases.</p>



<p class="wp-block-paragraph">Read that as: PHI detection is not de-identification. Under the HIPAA Privacy Rule, de-identification has two defined routes, Safe Harbor and Expert Determination. An NLP model with confidence scores is an input to either, not a substitute for either. If your analytics tier is meant to hold de-identified data, someone has to own that determination, and it will not be your pipeline.</p>



<p class="wp-block-paragraph">What works structurally:</p>



<ul class="wp-block-list">
<li>Put the analytics tier in a <strong>separate AWS account</strong> with its own KMS key. Not a separate bucket. A separate account, so the blast radius argument is enforced by IAM boundaries rather than by convention.</li>

<li>Make the flow <strong>one directional</strong>. The de-identification job reads from the identified side and writes to the analytics side. Nothing in the analytics account holds permissions pointing back.</li>

<li><strong>Replace identifiers with pseudonyms</strong> rather than deleting them, keeping the crosswalk in the identified account. This preserves your ability to join across results and to re-identify under a documented process, which the research use case usually needs.</li>

<li>Watch <strong>quasi-identifiers</strong>. Rare test panels, unusual reference ranges and precise timestamps re-identify people in small populations even after the obvious fields are gone. This is a statistics problem, not an IAM problem.</li>

<li>Use <strong>Lake Formation</strong> or equivalent for column and row filtering if analysts need partial access, so the restriction lives with the catalog rather than in each query tool.</li>
</ul>



<h2 class="wp-block-heading">Boundary five: sending results back out</h2>



<p class="wp-block-paragraph">Delivery is where clinical correctness and security pull against each other, and security usually loses quietly.</p>



<p class="wp-block-paragraph">The correction problem is the sharp edge. HL7 result messages carry a status that distinguishes preliminary, final and corrected results. If your delivery layer is idempotent on message ID alone, a corrected result with a new ID for the same observation will happily land beside the original instead of superseding it. Two conflicting values, both marked delivered, nobody alerted.</p>



<p class="wp-block-paragraph">Build the idempotency key from the identity of the <em>observation</em>, which typically means the placer or filler order identifier combined with the specific observation identifier, and carry the result status as a first-class field so a correction overwrites rather than appends. Keep every version, expose the current one.</p>



<p class="wp-block-paragraph">For the outbound path itself:</p>



<ul class="wp-block-list">
<li>Presigned URLs are convenient for report PDFs and they are also a bearer credential. Anyone holding the link has the document. Keep expiry short and treat generation as an auditable event.</li>

<li>If you expose a provider-facing portal, put a WAF in front of it. AWS WAF and Cloudflare are both credible here; the deciding factor is usually where the rest of your edge already lives, not the rule engines.</li>

<li>Never email a result. Email a notification that a result is available, behind authentication. The distinction sounds pedantic right up until someone forwards a mailbox.</li>
</ul>



<h2 class="wp-block-heading">Boundary six: deletion, and the conflict nobody plans for</h2>



<p class="wp-block-paragraph">Two obligations point in opposite directions. Retention rules say keep the record. Privacy rights and internal policy say be able to remove data. If you set compliance-mode Object Lock across everything, you have chosen one side without noticing.</p>



<p class="wp-block-paragraph">Resolve it by classifying before you lock. Records that fall under laboratory retention obligations go into the immutable tier with a retention period derived from the applicable rule. Derived artifacts, caches, intermediate parquet, analytics extracts and enrichment outputs do not belong there. They belong in buckets with lifecycle rules, and they are what you actually delete.</p>



<p class="wp-block-paragraph">Crypto shredding covers the middle ground: if a data set is encrypted under a key used for nothing else, scheduling that key for deletion renders the ciphertext unrecoverable without touching the objects. It is clean, it is verifiable, and it only works if you planned the key granularity up front. Retrofitting per-tenant or per-cohort keys onto a pipeline that used one key for everything is a full re-encryption exercise.</p>



<p class="wp-block-paragraph">Do not forget the endpoints. Instrument workstations, the interface engine box, and the laptop somebody used to test the parser all accumulate copies. Cloud controls do nothing for physical media, and secure erase tooling such as O&amp;O SafeErase exists for exactly that step in a decommissioning runbook.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting a medical lab data pipeline on AWS</h2>



<h3 class="wp-block-heading">Messages arrive but nothing appears downstream</h3>



<p class="wp-block-paragraph">Check the DLQ first, then check whether the DLQ itself has a consumer. A surprising number of pipelines have a correctly configured dead letter queue that nobody monitors, so failures accumulate silently and the only symptom is a gap in the data. Alarm on DLQ depth greater than zero, not on a threshold.</p>



<h3 class="wp-block-heading">Access denied from a Lambda that has the right IAM policy</h3>



<p class="wp-block-paragraph">With a customer managed KMS key, IAM permission on S3 is only half the grant. The function&#8217;s role also needs decrypt permission on the key, and the key policy has to allow it. If you are crossing accounts, both the key policy and the role policy must permit the action. This is the intended behavior of the two-lock design, and it is the single most common support question on any pipeline built this way.</p>



<h3 class="wp-block-heading">Deletion fails on an object you are certain you should be able to delete</h3>



<p class="wp-block-paragraph">Check for a legal hold before you check retention. A legal hold has no expiry and is independent of the retention period, so an object can be past its retention date and still undeletable. Also confirm which mode the retention uses, because governance and compliance produce the same error to a caller without bypass permission.</p>



<h3 class="wp-block-heading">Counts do not reconcile with the LIS</h3>



<p class="wp-block-paragraph">Compare on observations, not messages. One message can carry several results, corrections create additional messages for the same observation, and a naive message count will diverge from the LIS by exactly the amount that matters clinically. Reconcile daily and alert on drift rather than investigating at quarter end.</p>



<h3 class="wp-block-heading">Timestamps drift between systems</h3>



<p class="wp-block-paragraph">HL7 v2 timestamps do not always carry a timezone offset, and analyzers are often set to local time with no daylight saving handling. Capture the offset at ingest from the source system configuration and store everything in UTC with the original string preserved. Recovering an ambiguous timestamp after the fact is unpleasant, and around a daylight saving transition it may not be possible at all.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating &#8220;HIPAA eligible&#8221; as &#8220;compliant.&#8221;</strong> Eligibility means AWS will cover the service under the BAA. Configuration is still entirely yours.</li>

<li><strong>Putting identifiers in object keys, queue names, or log group names.</strong> Encrypted contents, plaintext metadata.</li>

<li><strong>Enabling compliance-mode Object Lock before validating retention values.</strong> There is no undo, and no support ticket that fixes it.</li>

<li><strong>Assuming automated PHI detection equals de-identification.</strong> AWS itself says the entity list does not map one to one to Safe Harbor identifiers.</li>

<li><strong>Running the analytics tier in the same account as identified data.</strong> A bucket boundary is a policy away from collapsing. An account boundary is not.</li>

<li><strong>Shipping application logs to an external platform without checking what the log lines contain.</strong> Test it by grepping your own log group for a known test patient name.</li>

<li><strong>Ignoring corrected results until an analyst notices.</strong> Amendment handling is a day-one requirement, not a phase two feature.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li><strong>Write the data flow diagram first and mark every trust boundary crossing.</strong> Every arrow that crosses one needs a named control. This exercise finds more problems than any scanner.</li>

<li><strong>Encode the eligible service list as policy.</strong> Service control policies at the organization level stop someone from putting PHI into a service you never assessed.</li>

<li><strong>Keep VPC endpoints on the PHI path.</strong> Interface and gateway endpoints keep traffic to AWS services off the public internet and give you an endpoint policy as an extra choke point. Check which of your chosen services offer one.</li>

<li><strong>Build with synthetic HL7 from day one.</strong> Nobody should need production PHI to develop a parser. A cheap VPS from Contabo or InterServer is fine for a synthetic-data development box, and keeping that environment entirely outside the PHI boundary is the point.</li>

<li><strong>Alarm on absence.</strong> A feed that stops is more dangerous than a feed that errors, because errors are loud. Alert when expected message volume for a source drops below its floor.</li>

<li><strong>Test restores, not backups.</strong> Restore into an isolated account, confirm the KMS grants work there, and document how long it took.</li>

<li><strong>Review CloudTrail for data events on the PHI buckets.</strong> Object-level logging costs money and is the only record of who read what.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do I need a BAA with AWS before I start building?</h3>



<p class="wp-block-paragraph">Before PHI touches the environment, yes. You can architect and test with synthetic data first. AWS provides the business associate addendum through AWS Artifact as a self-service acceptance that can be applied across an organization, and only services listed on the HIPAA Eligible Services Reference may process, store or transmit PHI.</p>



<h3 class="wp-block-heading">Should I use AWS HealthLake or build on S3, Glue and Athena?</h3>



<p class="wp-block-paragraph">HealthLake is a managed FHIR datastore with FHIR APIs, medical NLP and query built in, and it is the shorter path if your consumers speak FHIR. It does not natively ingest HL7 v2; AWS points to partner tooling or your own transformation for non-FHIR input. If your consumers are analysts with SQL and the data is structured lab results rather than clinical documents, an S3 and Athena lakehouse is simpler and easier to keep inside a boundary you control. Decide on the consumer, not the source.</p>



<h3 class="wp-block-heading">How long do I have to keep laboratory records?</h3>



<p class="wp-block-paragraph">For CLIA-regulated laboratories in the United States, 42 CFR 493.1105 sets the federal floor, including at least two years for test reports after the date of reporting and at least ten years for pathology test reports. State law frequently exceeds these minimums, so the applicable period is whichever is longer for your jurisdiction. Read the regulation and confirm with your compliance lead before you encode a number into a retention policy.</p>



<h3 class="wp-block-heading">Can Amazon Comprehend Medical de-identify data for me?</h3>



<p class="wp-block-paragraph">It can detect PHI entities and give you confidence scores, which is a strong starting point for redaction workflows. It does not by itself satisfy HIPAA&#8217;s de-identification standard. AWS documents that the detected entities do not map one to one to the Safe Harbor identifier list and recommends human review for compliance use cases. Treat it as a detector feeding a documented Safe Harbor or Expert Determination process.</p>



<h3 class="wp-block-heading">Is S3 Object Lock compliance mode required for lab records?</h3>



<p class="wp-block-paragraph">Not inherently. The argument for compliance mode is a threat model in which a compromised or malicious administrator could remove retention, not a regulation that names the mode. Governance mode plus tightly controlled bypass permissions and strong audit logging is a defensible position for many laboratories. Choose deliberately and write down the reasoning, because compliance mode commits you to the storage cost for the full term with no exit.</p>



<h3 class="wp-block-heading">Where does PHI most often leak in a pipeline that looks correctly configured?</h3>



<p class="wp-block-paragraph">Logs, dead letter queues, workflow execution histories, notification messages and object key names. All five are metadata surfaces that sit outside the storage layer everyone reviews. Grep your own log groups for a known test patient identifier and you will find out quickly whether your pipeline has the problem.</p>



<h3 class="wp-block-heading">Do I need a dedicated AWS account just for the lab pipeline?</h3>



<p class="wp-block-paragraph">Separate accounts for identified and de-identified data are worth it even at small scale, because the boundary is then enforced by IAM rather than by naming conventions. Whether the PHI workload also needs isolating from your other production workloads depends on who holds admin access there. If the answer is &#8220;broadly the same people,&#8221; separate it.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A secure medical lab data pipeline on AWS is not defined by the encryption on its storage layer. Encryption is the easy part, and it is the part everyone gets right. What separates a design that survives an audit from one that does not is whether PHI can travel by value through your operational plane: into logs, into queue bodies, into workflow histories, into filenames, into alert emails.</p>



<p class="wp-block-paragraph">Move PHI by reference, keep identifiers out of every metadata surface, and put an account boundary between identified and de-identified data. Do those three things and the rest of the compliance work becomes documentation rather than redesign.</p>



<h2 class="wp-block-heading">Work with me on healthcare data pipelines</h2>



<p class="wp-block-paragraph">I design and review AWS data platforms for regulated workloads, and lab data is a specific enough problem that generic cloud advice tends to miss it. Things I can help with:</p>



<ul class="wp-block-list">
<li>Mapping every trust boundary in an existing pipeline and finding the PHI surfaces nobody documented, including logs, queues and object key schemes.</li>

<li>Designing HL7 v2 or file-drop ingestion into S3 with KMS key separation, VPC endpoints and bucket policies that enforce rather than suggest.</li>

<li>Building the identified and de-identified account split, including the one-way transformation job and the pseudonym crosswalk.</li>

<li>Getting amendment and correction handling right so corrected results supersede rather than duplicate, all the way to the analytics tables.</li>

<li>Retention and deletion design: Object Lock modes, lifecycle policies, crypto shredding key granularity, and the runbook that ties them together.</li>

<li>Observability that is useful without being a disclosure risk, including CloudWatch retention hygiene and safe alerting patterns.</li>
</ul>



<p class="wp-block-paragraph">If you have an architecture diagram, a Terraform plan, or a redacted sample of the messages you are receiving, send it over and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/medical-lab-data-pipeline-aws/">Where PHI Escapes: Building a Secure Medical Lab Data Pipeline on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation</title>
		<link>https://john-nessime.com/blog/devops/docker-log-rotation/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 10 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Docker]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Container Runtime]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Disk Space]]></category>
		<category><![CDATA[Docker Compose]]></category>
		<category><![CDATA[journald]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[Logs]]></category>
		<category><![CDATA[Loki]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[rsyslog]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[Storage]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Syslog]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=182</guid>

					<description><![CDATA[<p>Docker's default logging driver writes container output to a JSON file with no size limit and no rotation. This is a practical guide to Docker log rotation: what the defaults actually do, why your daemon.json change did nothing, why deleting the log file did not free any disk, and how to choose between json-file, local, journald and shipping logs off the box.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/docker-log-rotation/">Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Monitoring says the API is healthy. The ticket says nobody can deploy. You SSH in, run <code>df -h</code>, and the root filesystem is at 100%. A few minutes of poking around <code>/var/lib/docker</code> later you find it: one container, one file, a double-digit number of gigabytes of newline-delimited JSON, written continuously since the day that container was created.</p>



<p class="wp-block-paragraph">Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker&#8217;s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.</p>



<p class="wp-block-paragraph">This post covers Docker log rotation properly: what the defaults actually do, why the <code>daemon.json</code> change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.</p>



<h2 class="wp-block-heading">The default is unbounded, and that is documented behaviour</h2>



<p class="wp-block-paragraph">Docker&#8217;s default logging driver is <code>json-file</code>. It captures stdout and stderr from the container&#8217;s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.</p>



<p class="wp-block-paragraph">The important part is the option defaults. For <code>json-file</code>, <code>max-size</code> defaults to unlimited, <code>max-file</code> defaults to 1, and <code>compress</code> defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.</p>



<p class="wp-block-paragraph">Docker&#8217;s own documentation is upfront about why. Keeping <code>json-file</code> unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.</p>



<p class="wp-block-paragraph">Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:</p>



<pre class="wp-block-code"><code># What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"</code></pre>



<p class="wp-block-paragraph">The gap between the first command and the second is where most of the pain lives, and we will come back to it.</p>



<p class="wp-block-paragraph">To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container&#8217;s log file plus a small amount of metadata, so the sizes are close enough to be useful:</p>



<pre class="wp-block-code"><code>sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10</code></pre>



<p class="wp-block-paragraph">Container IDs are not memorable, so map the winner back to a name with <code>docker ps --no-trunc</code> or by grepping the ID against <code>docker ps -aq</code>. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.</p>



<h2 class="wp-block-heading">Setting up Docker log rotation on the daemon</h2>



<p class="wp-block-paragraph">The daemon-wide fix goes in <code>/etc/docker/daemon.json</code>. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.</p>



<pre class="wp-block-code"><code>{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}</code></pre>



<p class="wp-block-paragraph">What each key buys you:</p>



<ul class="wp-block-list">
<li><code>max-size</code> caps a single file. Accepts an integer plus <code>k</code>, <code>m</code> or <code>g</code>. This is the setting that stops the runaway.</li>

<li><code>max-file</code> caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is <em>only effective if <code>max-size</code> is also set</em>, which is the single most common misconfiguration I see.</li>

<li><code>compress</code> gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.</li>
</ul>



<p class="wp-block-paragraph">Two things trip people up here. First, every value in <code>log-opts</code> must be a JSON string, including the numeric ones. Writing <code>"max-file": 3</code> without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed <code>daemon.json</code> on a production host is a much worse afternoon than a full disk:</p>



<pre class="wp-block-code"><code># Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  &amp;&amp; echo "daemon.json parses OK"

sudo systemctl restart docker</code></pre>



<p class="wp-block-paragraph">Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is <code>max-size</code> multiplied by <code>max-file</code>. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.</p>



<h2 class="wp-block-heading">Why your daemon.json change did nothing</h2>



<p class="wp-block-paragraph">This is the failure mode that costs the most time, because it looks exactly like the fix not working.</p>



<p class="wp-block-paragraph">Logging configuration is baked into a container&#8217;s host config when the container is <em>created</em>. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because <code>docker restart</code> stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.</p>



<p class="wp-block-paragraph">So the container that filled your disk keeps filling your disk, its log file still has no cap, and <code>docker info</code> cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.</p>



<p class="wp-block-paragraph">To actually apply it, recreate the container and then verify rather than assuming:</p>



<pre class="wp-block-code"><code># Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web</code></pre>



<p class="wp-block-paragraph">You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.</p>



<p class="wp-block-paragraph">Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.</p>



<h3 class="wp-block-heading">Pinning limits per service in Compose</h3>



<p class="wp-block-paragraph">Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides <code>daemon.json</code> entirely. A YAML anchor keeps it from turning into copy-paste sprawl:</p>



<pre class="wp-block-code"><code>x-logging: &amp;default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"</code></pre>



<p class="wp-block-paragraph">Be aware that the <code>logging</code> block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with <em>its</em> defaults, which for <code>json-file</code> means straight back to unbounded. Half-specifying is worse than not specifying at all.</p>



<h2 class="wp-block-heading">You deleted the log file and df did not move</h2>



<p class="wp-block-paragraph">Classic 3am mistake, and it is worth understanding rather than memorising.</p>



<p class="wp-block-paragraph">When you <code>rm</code> the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from <code>ls</code>, the daemon keeps writing into a file you can no longer see, and <code>df</code> reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.</p>



<p class="wp-block-paragraph"><code>truncate -s 0</code> is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:</p>



<pre class="wp-block-code"><code>sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"</code></pre>



<p class="wp-block-paragraph">There is a cost. Truncating out from under the daemon can leave an in-flight <code>docker logs -f</code> stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.</p>



<p class="wp-block-paragraph">The same reasoning applies to <code>logrotate</code> rules pointed at <code>/var/lib/docker/containers/*/*.log</code>. Rotating with <code>copytruncate</code> works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker&#8217;s own docs warn against external tools touching those files. If the driver can rotate for you, let it.</p>



<h2 class="wp-block-heading">Choosing a driver, and what each one costs</h2>



<h3 class="wp-block-heading">json-file</h3>



<p class="wp-block-paragraph">The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. <code>docker logs</code> works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.</p>



<h3 class="wp-block-heading">local</h3>



<p class="wp-block-paragraph">Docker&#8217;s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it <em>rotates and compresses by default</em>, retaining five files of 20 MB each per container out of the box. <code>docker logs</code> works exactly as normal. The trade-off is that the format is Docker&#8217;s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is <code>docker logs</code> plus a shipper that talks to the daemon, this is the safest default you can pick.</p>



<h3 class="wp-block-heading">journald</h3>



<p class="wp-block-paragraph">Hands container output to systemd&#8217;s journal, which means you get <code>journalctl</code> filtering, structured fields, and the journal&#8217;s own size management via <code>SystemMaxUse</code> in <code>journald.conf</code>. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.</p>



<p class="wp-block-paragraph">The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at <code>RateLimitIntervalSec</code> and <code>RateLimitBurst</code> before you need them, not after.</p>



<h3 class="wp-block-heading">syslog, fluentd and the remote drivers</h3>



<p class="wp-block-paragraph">These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.</p>



<p class="wp-block-paragraph">What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.</p>



<h3 class="wp-block-heading">none</h3>



<p class="wp-block-paragraph">Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because <code>docker logs</code> returns nothing and you will waste twenty minutes concluding the container is broken.</p>



<h3 class="wp-block-heading">The blocking behaviour worth knowing about</h3>



<p class="wp-block-paragraph">By default the logging path is blocking. If the driver cannot keep up, the container&#8217;s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.</p>



<pre class="wp-block-code"><code>docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1</code></pre>



<p class="wp-block-paragraph">Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.</p>



<h3 class="wp-block-heading">Dual logging, and why docker logs still works</h3>



<p class="wp-block-paragraph">If you switch to a remote driver and find <code>docker logs</code> still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the <code>local</code> driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.</p>



<p class="wp-block-paragraph">Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Reclaiming space when the disk is already full</h2>



<p class="wp-block-paragraph">Order matters here. Free space first, fix the cause second, verify third.</p>



<ol class="wp-block-list">
<li>Confirm logs are actually the problem. Compare <code>du -sh /var/lib/docker/containers</code> against <code>docker system df</code>, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.</li>

<li>Identify the largest log files with the <code>du</code> and <code>sort</code> command from earlier.</li>

<li>Copy anything you need out first, if the logs are part of an active investigation.</li>

<li>Truncate the offenders with <code>truncate -s 0</code>. Never <code>rm</code>.</li>

<li>Set the limits in <code>daemon.json</code>, validate the JSON, restart the daemon.</li>

<li>Recreate the containers so the limits take effect, then confirm with <code>docker inspect</code> on each one.</li>

<li>Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.</li>
</ol>



<p class="wp-block-paragraph">Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>Rotation is configured but files still grow past the cap.</strong> The container predates the change. Check <code>docker inspect --format '{{.HostConfig.LogConfig}}'</code> and recreate it.</p>



<p class="wp-block-paragraph"><strong>The daemon will not start after editing daemon.json.</strong> Almost always invalid JSON, or unquoted values in <code>log-opts</code>. Read the actual error with <code>journalctl -u docker --no-pager -n 50</code> rather than guessing.</p>



<p class="wp-block-paragraph"><strong>max-file is set but only one file ever appears.</strong> <code>max-file</code> does nothing without <code>max-size</code>. Set both.</p>



<p class="wp-block-paragraph"><strong>docker logs -f hangs after showing existing output.</strong> Something truncated the file underneath the daemon, usually an external <code>logrotate</code> rule. Reattach to recover, then remove the rule.</p>



<p class="wp-block-paragraph"><strong>docker logs returns nothing at all.</strong> Check the driver. <code>none</code> discards everything, and some remote drivers only serve reads through the dual-logging cache.</p>



<p class="wp-block-paragraph"><strong>Disk usage did not drop after deleting log files.</strong> The daemon still holds the descriptors. Restart the daemon to release the inodes, and use <code>truncate</code> next time.</p>



<p class="wp-block-paragraph"><strong>Logs vanish sooner than the retention you configured.</strong> On <code>journald</code>, check journal rate limiting and <code>SystemMaxUse</code>. On a remote driver in non-blocking mode, check whether the buffer is overflowing.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Assuming a daemon restart applies the new limits to running containers. It does not.</li>

<li>Setting <code>max-file</code> without <code>max-size</code> and believing rotation is on.</li>

<li>Writing numeric values unquoted in <code>log-opts</code>.</li>

<li>Overwriting an existing <code>daemon.json</code> instead of merging, and silently dropping storage or network settings.</li>

<li>Using <code>rm</code> on a live container&#8217;s log file and concluding the disk report is broken.</li>

<li>Specifying a <code>logging.driver</code> in Compose without its <code>options</code>, which resets that service back to the driver&#8217;s own unbounded defaults.</li>

<li>Pointing host <code>logrotate</code> at the Docker containers directory when the driver could have handled it.</li>

<li>Sizing rotation per container without multiplying by container count against the actual disk.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Set logging limits in <code>daemon.json</code> as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.</li>

<li>Prefer the <code>local</code> driver on standalone hosts unless something specifically needs to read the JSON files.</li>

<li>Turn on <code>compress</code>. Log text compresses well enough that you get more retention for the same disk.</li>

<li>Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.</li>

<li>Do the multiplication. Worst-case host usage is <code>max-size</code> × <code>max-file</code> × container count, and it should be a fraction of your disk, not most of it.</li>

<li>Alert on Docker data root usage at a threshold that leaves you time to act.</li>

<li>Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.</li>

<li>Verify with <code>docker inspect</code> after any change. <code>docker info</code> tells you the daemon&#8217;s intent, not what your containers carry.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does Docker rotate container logs automatically?</h3>



<p class="wp-block-paragraph">Not with the default driver. <code>json-file</code> ships with <code>max-size</code> unlimited and <code>max-file</code> set to 1, so a single file grows without bound. The <code>local</code> driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.</p>



<h3 class="wp-block-heading">Where are Docker container logs stored on the host?</h3>



<p class="wp-block-paragraph">Under the Docker data root, one directory per container, which on a standard Linux install means <code>/var/lib/docker/containers/</code>. Rather than assembling the path by hand, ask the daemon: <code>docker inspect --format '{{.LogPath}}' &lt;container&gt;</code>. That works regardless of a custom <code>data-root</code>.</p>



<h3 class="wp-block-heading">How do I clear Docker logs without stopping the container?</h3>



<p class="wp-block-paragraph">Truncate the file to zero bytes with <code>truncate -s 0</code> on the path from <code>{{.LogPath}}</code>. Do not use <code>rm</code>: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.</p>



<h3 class="wp-block-heading">Why did my daemon.json rotation settings not apply?</h3>



<p class="wp-block-paragraph">Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and <code>docker restart</code> reuses the same container object. Recreate the container, then confirm with <code>docker inspect --format '{{.HostConfig.LogConfig}}'</code>.</p>



<h3 class="wp-block-heading">Should I use the local driver or json-file?</h3>



<p class="wp-block-paragraph">Use <code>local</code> if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use <code>json-file</code> if an agent or shipper on the host tails those files, because it cannot parse the <code>local</code> format. Either way <code>docker logs</code> behaves the same.</p>



<h3 class="wp-block-heading">Can I use logrotate for Docker container logs instead?</h3>



<p class="wp-block-paragraph">You can, and plenty of hosts do, but the driver options are the better tool. A <code>copytruncate</code> rule pointed at the containers directory reaches into files the daemon treats as private, and can leave <code>docker logs -f</code> streams stalled after each rotation. Use <code>logrotate</code> for logs your applications write to mounted volumes, and let the driver handle stdout.</p>



<h3 class="wp-block-heading">How much disk should I budget for container logs?</h3>



<p class="wp-block-paragraph">Worst case is <code>max-size</code> × <code>max-file</code> per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.</p>



<p class="wp-block-paragraph">So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with <code>docker inspect</code> rather than trusting <code>docker info</code>. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with container logging on your hosts?</h2>



<p class="wp-block-paragraph">Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:</p>



<ul class="wp-block-list">
<li>Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits</li>

<li>Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number</li>

<li>Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back</li>

<li>Choosing between <code>json-file</code>, <code>local</code>, <code>journald</code> and shipping off-box, including the blocking behaviour trade-offs under load</li>

<li>Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends</li>

<li>Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me your <code>daemon.json</code>, the output of <code>docker ps</code> and a <code>du -sh</code> of your containers directory, and I will tell you what I would change.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/docker-log-rotation/">Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>journald vs rsyslog: Where Your Linux Logs Actually Live</title>
		<link>https://john-nessime.com/blog/devops/journald-vs-rsyslog-where-logs-live/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 08 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Disk Space]]></category>
		<category><![CDATA[journalctl]]></category>
		<category><![CDATA[journald]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[Logs]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[rsyslog]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Syslog]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=168</guid>

					<description><![CDATA[<p>On a stock Linux server, journald and rsyslog are both running and both storing the same events, with retention policies that don't know about each other. Here's where each log line actually lives, the three failure modes that bite in production, and how to decide which copy you keep.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/journald-vs-rsyslog-where-logs-live/">journald vs rsyslog: Where Your Linux Logs Actually Live</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The disk alert usually comes first. <code>/var</code> sitting at 94 percent on a box that runs three services and logs almost nothing interesting. You go digging and find the same night&#8217;s SSH failures in two places: compressed binary files under <code>/var/log/journal/</code>, and plain text in <code>/var/log/secure</code>. Same events. Two copies. Nobody decided that. It is just what the distro shipped.</p>



<p class="wp-block-paragraph">The worse version of this problem is the mirror image. You reboot a server to clear something, come back to read what led up to it, and the journal starts at the reboot. Everything before is gone. Not rotated, not compressed, not archived somewhere clever. Gone, because the journal was living in RAM and nobody ever told it otherwise.</p>



<p class="wp-block-paragraph">Both of those are the same misunderstanding wearing different clothes: not knowing where your logs actually live. This post walks through the <strong>journald vs rsyslog</strong> split on a modern Linux server, what each one really stores, the three failure modes that bite in production, and how to decide which of the two is the copy you keep.</p>



<h2 class="wp-block-heading">What actually happens when a process logs a line</h2>



<p class="wp-block-paragraph">On any systemd-based distribution, <code>systemd-journald</code> is the first stop for almost everything. It is not one option among several. It sits underneath.</p>



<ul class="wp-block-list">
<li>Anything a unit writes to stdout or stderr, because systemd wires those to the journal by default.</li>

<li>Kernel messages, read from the kernel ring buffer.</li>

<li>Classic <code>syslog(3)</code> calls from libc, which land on the <code>/dev/log</code> socket that journald owns.</li>

<li>Native journal API calls from anything linked against libsystemd, which carry structured key-value fields instead of a flat string.</li>

<li>Audit records, if <code>Audit=yes</code> is in effect.</li>
</ul>



<p class="wp-block-paragraph">journald writes all of that into indexed binary journal files. Then, on most server distributions, rsyslog gets a <em>second copy</em> of the same events and writes them out as text into <code>/var/log/messages</code>, <code>/var/log/secure</code>, <code>/var/log/maillog</code> and friends.</p>



<p class="wp-block-paragraph">That handoff happens one of two ways, and which one your box uses matters more than most people expect. More on that below. First, the part that costs you an outage.</p>



<p class="wp-block-paragraph">Before changing anything, look at the effective journald configuration rather than the file you think is authoritative. Distributions ship drop-ins under <code>/usr/lib/systemd/journald.conf.d/</code> that quietly override upstream defaults:</p>



<pre class="wp-block-code"><code># Print the merged configuration, including every drop-in, in load order
systemd-analyze cat-config systemd/journald.conf</code></pre>



<p class="wp-block-paragraph">This is the single most useful command in this whole post. Half the arguments about &#8220;what the default is&#8221; evaporate once you run it, because upstream systemd and your distribution frequently disagree about <code>ForwardToSyslog</code>.</p>



<h2 class="wp-block-heading">Failure one: the journal that was never on disk</h2>



<p class="wp-block-paragraph">journald&#8217;s <code>Storage=</code> setting defaults to <code>auto</code>. That word does more work than it looks like. Under <code>auto</code>, journald writes to <code>/var/log/journal/</code> <em>only if that directory already exists</em>. If it does not, journald falls back to <code>/run/log/journal/</code>, which is tmpfs. Memory. Wiped on reboot.</p>



<p class="wp-block-paragraph">Nothing warns you. <code>journalctl</code> works fine, colours are pretty, filters work. You just silently have no history.</p>



<p class="wp-block-paragraph">The quickest test is boot history. If the journal is persistent you see multiple boots; if it is volatile you see exactly one:</p>



<pre class="wp-block-code"><code>journalctl --list-boots</code></pre>



<p class="wp-block-paragraph">To make it persistent, create the directory, let systemd-tmpfiles apply the correct ownership and ACLs, then restart the daemon and flush anything still held in RAM:</p>



<pre class="wp-block-code"><code>sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush
journalctl --list-boots</code></pre>



<p class="wp-block-paragraph">The <code>systemd-tmpfiles</code> step is the one people skip. The journal directory needs specific group ownership and ACLs so that unprivileged users can read their own entries and members of the <code>systemd-journal</code> group can read everything. Creating the directory with a bare <code>mkdir</code> and walking away usually works, but it is the kind of &#8220;usually&#8221; that produces a confusing permissions ticket six months later.</p>



<p class="wp-block-paragraph">If you want the behaviour to be explicit rather than inferred from a directory&#8217;s existence, set it in <code>/etc/systemd/journald.conf.d/00-storage.conf</code>:</p>



<pre class="wp-block-code"><code>[Journal]
Storage=persistent</code></pre>



<p class="wp-block-paragraph">A drop-in file is better than editing the main config, because package upgrades will not fight you over it.</p>



<h2 class="wp-block-heading">Failure two: paying for the same log line twice</h2>



<p class="wp-block-paragraph">Once the journal is persistent, you are storing everything twice on a default server build: once as compressed binary in the journal, once as text under <code>/var/log/</code>. Two stores, two completely separate retention policies, neither of which knows the other exists.</p>



<p class="wp-block-paragraph">The journal is capped by <code>SystemMaxUse=</code> and <code>SystemKeepFree=</code>. Left unset, they default to 10 percent and 15 percent of the filesystem respectively, and journald honours whichever is stricter. Recent systemd also caps those calculated defaults so they do not grow without bound on a large disk, which is worth knowing if you have ever wondered why a huge volume did not produce a proportionally huge journal.</p>



<p class="wp-block-paragraph">The text copy is capped by <code>logrotate</code>, configured somewhere completely different, usually <code>/etc/logrotate.d/rsyslog</code>, on a weekly or daily schedule with its own <code>rotate</code> count.</p>



<p class="wp-block-paragraph">Measure both before you tune either:</p>



<pre class="wp-block-code"><code># What the journal is actually consuming on disk
journalctl --disk-usage

# What everything under /var/log costs, sorted, biggest last
sudo du -sh /var/log/* | sort -h</code></pre>



<p class="wp-block-paragraph">On a small VPS this stops being academic quickly. A modest instance from a provider like InterServer, Hetzner or DigitalOcean often has <code>/var</code> sharing a single root volume of 20 to 40 GB. A journal allowed to take 10 percent of that, plus four weeks of rotated text logs, plus a database, plus a container image cache, is how you end up paged at two in the morning for something that is not a real incident.</p>



<p class="wp-block-paragraph">If you decide the text files are the copy you keep, put the journal on a short leash:</p>



<pre class="wp-block-code"><code>[Journal]
# Hard ceiling on total journal size
SystemMaxUse=200M
# Never let the journal push free space below this
SystemKeepFree=1G
# Discard entries older than this regardless of size
MaxRetentionSec=2day</code></pre>



<p class="wp-block-paragraph">Those three do different jobs and you generally want at least two of them. <code>SystemMaxUse</code> bounds the journal itself. <code>SystemKeepFree</code> protects the rest of the filesystem from the journal. <code>MaxRetentionSec</code> bounds it in time, which is what auditors and data-retention policies actually care about.</p>



<p class="wp-block-paragraph">To reclaim space immediately without waiting for the next rotation:</p>



<pre class="wp-block-code"><code># Trim archived journal files until the total falls below 500M
sudo journalctl --vacuum-size=500M

# Or trim by age instead
sudo journalctl --vacuum-time=7d</code></pre>



<p class="wp-block-paragraph">One caveat that surprises people: vacuuming only removes <em>archived</em> journal files. The currently active file is never deleted. If almost all your usage is in one large active file, run <code>journalctl --rotate</code> first, then vacuum.</p>



<h2 class="wp-block-heading">Failure three: two rate limiters, both silent</h2>



<p class="wp-block-paragraph">This is the one that is invisible until it matters, and it is the real reason to understand the journald vs rsyslog relationship rather than treating them as interchangeable.</p>



<p class="wp-block-paragraph">There are <em>two independent rate limiters</em> in the default pipeline, and both drop messages quietly.</p>



<h3 class="wp-block-heading">journald&#8217;s limiter</h3>



<p class="wp-block-paragraph">journald applies <code>RateLimitIntervalSec=</code> and <code>RateLimitBurst=</code> per service, defaulting to 30 seconds and 10000 messages. Exceed the burst inside the interval and the rest of that service&#8217;s messages in that window are discarded. Not queued. Discarded.</p>



<p class="wp-block-paragraph">The one piece of good news is that journald tells you, in the journal itself. Grep for it:</p>



<pre class="wp-block-code"><code>journalctl --grep="Suppressed" -n 50 --no-pager</code></pre>



<p class="wp-block-paragraph">A chatty web server or a container using the journald log driver will hit this. Rather than disabling rate limiting globally, override it for the one unit that needs it, using a drop-in:</p>



<pre class="wp-block-code"><code># /etc/systemd/system/nginx.service.d/logging.conf
[Service]
LogRateLimitIntervalSec=30s
LogRateLimitBurst=50000</code></pre>



<p class="wp-block-paragraph">Per-unit values override <code>journald.conf</code> for that unit only, which is exactly what you want. Turning the global limiter off means one runaway process can fill your disk and take the box down. Turning it up for the single service you know is noisy is a bounded decision.</p>



<h3 class="wp-block-heading">rsyslog&#8217;s limiter</h3>



<p class="wp-block-paragraph">If rsyslog is reading via the <code>imjournal</code> module, it applies <em>its own</em> rate limit on top, defaulting to 20000 messages per 600 seconds. A message can therefore survive journald&#8217;s limiter, land in the journal, and still never reach <code>/var/log/messages</code> or your remote log host.</p>



<p class="wp-block-paragraph">That is the nasty case, because your local investigation finds the events but your centralised alerting never fired. Two people look at two sources and reach opposite conclusions about whether something happened.</p>



<p class="wp-block-paragraph">rsyslog exposes counters for this through its statistics module, including how many messages it read from the journal, how many it submitted onward, and how many it discarded to rate limiting. Enable <code>impstat</code> if you run anything log-volume-sensitive; the discard counter is the number that tells you whether your pipeline is lying to you.</p>



<h2 class="wp-block-heading">How rsyslog gets its copy: imjournal or imuxsock</h2>



<p class="wp-block-paragraph">There are two mechanisms, they behave differently under load, and running both at once is a classic source of duplicate lines.</p>



<h3 class="wp-block-heading">imuxsock: journald pushes a copy to a socket</h3>



<p class="wp-block-paragraph">With <code>ForwardToSyslog=yes</code> in journald, journald writes a classic syslog-formatted copy of each message to a dedicated socket, and rsyslog reads it with <code>imuxsock</code>.</p>



<ul class="wp-block-list">
<li><strong>Where it wins:</strong> simple, no state file, no second rate limiter, and no possibility of rsyslog reading back its own output and looping.</li>

<li><strong>Where it does not:</strong> you get the flat syslog view only. Structured journal fields such as the originating unit are not carried across.</li>
</ul>



<h3 class="wp-block-heading">imjournal: rsyslog pulls from the journal</h3>



<p class="wp-block-paragraph"><code>imjournal</code> reads journal files directly and keeps a state file so it can resume where it left off after a restart.</p>



<ul class="wp-block-list">
<li><strong>Where it wins:</strong> structured fields survive the trip. If you want to route or template on the systemd unit name rather than parsing it out of a message string, this is the only option of the two.</li>

<li><strong>Where it does not:</strong> extra rate limiter to reason about, a state file that can drift, and a documented risk that a corrupted journal database causes rsyslog to re-read the same entries in a loop. rsyslog&#8217;s own documentation recommends using <code>imuxsock</code> instead unless you specifically need the structured data.</li>
</ul>



<p class="wp-block-paragraph">A representative <code>imjournal</code> load line, with the rate limit stated explicitly rather than left implicit:</p>



<pre class="wp-block-code"><code>module(load="imjournal"
       StateFile="imjournal.state"
       Ratelimit.Interval="600"
       Ratelimit.Burst="20000")</code></pre>



<p class="wp-block-paragraph">Writing the defaults out by hand is worth the two extra lines. The next person to debug missing messages will see the limiter exists instead of assuming there isn&#8217;t one.</p>



<p class="wp-block-paragraph">To find out which mechanism your box is using right now:</p>



<pre class="wp-block-code"><code>grep -R "imjournal|imuxsock" /etc/rsyslog.conf /etc/rsyslog.d/ 2&gt;/dev/null
grep -R "ForwardToSyslog" /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ /usr/lib/systemd/journald.conf.d/ 2&gt;/dev/null</code></pre>



<p class="wp-block-paragraph">If the first command returns both modules and the second returns <code>ForwardToSyslog=yes</code>, you have a duplication problem. Pick one path and disable the other.</p>



<h2 class="wp-block-heading">Only one of the two leaves the box</h2>



<p class="wp-block-paragraph">This is the argument that settles most journald vs rsyslog debates in practice. journald is a local store. In a stock server install it has no mechanism for shipping logs to another host. There are separate systemd components for journal upload and reception, and third-party projects that read the journal and forward it, but none of that is in the default path.</p>



<p class="wp-block-paragraph">rsyslog forwards as a first-class feature, and it does the part that actually matters: it buffers when the destination is unreachable.</p>



<pre class="wp-block-code"><code>global(workDirectory="/var/spool/rsyslog")

action(type="omfwd"
       target="logs.example.net"
       port="514"
       protocol="tcp"
       action.resumeRetryCount="-1"
       queue.type="LinkedList"
       queue.filename="fwd_loghost"
       queue.maxDiskSpace="1g"
       queue.saveOnShutdown="on")</code></pre>



<p class="wp-block-paragraph">Line by line, because these are not decoration:</p>



<ul class="wp-block-list">
<li><code>workDirectory</code> is where spool files get written. Without it, the disk queue has nowhere to go.</li>

<li><code>protocol="tcp"</code> rather than UDP. UDP syslog silently drops under congestion, which defeats the point of forwarding at all.</li>

<li><code>queue.type="LinkedList"</code> makes the action asynchronous so a slow destination does not block local processing.</li>

<li><code>queue.filename</code> is what actually enables disk assistance. In-memory queue first, spilling to disk when it fills. The name must be unique across every action in the config.</li>

<li><code>queue.maxDiskSpace</code> caps that spool. Set it, or a long outage at the far end fills the disk you were trying to protect.</li>

<li><code>queue.saveOnShutdown="on"</code> persists whatever is still queued across a service restart.</li>
</ul>



<p class="wp-block-paragraph">One honest caveat on <code>action.resumeRetryCount="-1"</code>. Infinite retry is the right default for &#8220;never lose a log line&#8221;, but there are reported cases where a relay under sustained back-pressure stops accepting new inbound messages while retrying forever. If your host is a relay rather than a leaf, test that behaviour deliberately with the destination firewalled off before you trust it.</p>



<p class="wp-block-paragraph">rsyslog is not the only way off the box. Grafana Alloy into Loki, Vector, and Fluent Bit all read the journal directly and speak modern backends, and hosted platforms such as Better Stack, Papertrail or Datadog will take a plain syslog stream if you would rather not run the storage side. The mechanism differs; the decision does not. Something has to own the write path off the machine, and journald on its own is not it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">journald vs rsyslog: how I would decide</h2>



<p class="wp-block-paragraph">Stop asking which is better. Ask which one is the copy you are willing to be judged on, then make the other one cheap.</p>



<ol class="wp-block-list">
<li><strong>Do logs need to leave this machine?</strong> If yes, rsyslog or a dedicated shipper owns the outbound path. That is settled before anything else.</li>

<li><strong>Do you troubleshoot mostly by unit?</strong> If <code>journalctl -u something -b</code> is your muscle memory, journald is your primary read path and should get the retention budget.</li>

<li><strong>Do existing tools read text files?</strong> Fail2ban, logwatch, older SIEM collectors and a lot of home-grown scripts parse <code>/var/log/*</code>. Turning rsyslog off breaks them quietly.</li>

<li><strong>How much disk do you actually have?</strong> Under about 40 GB, keeping two full copies is an unforced error. Cap one hard.</li>

<li><strong>Is tamper-evidence a requirement?</strong> journald supports Forward Secure Sealing, which detects after-the-fact modification of journal files. Shipping to an append-only remote store is the more commonly accepted answer, and the two are not mutually exclusive.</li>
</ol>



<p class="wp-block-paragraph">The configuration I reach for first on a general-purpose server: journald persistent with a hard cap for a few days of fast local querying, rsyslog kept for the syslog files that other tools expect plus reliable forwarding off-box, <code>imuxsock</code> as the handoff unless something genuinely needs structured fields, and logrotate retention trimmed down because the remote copy is the one that matters for anything older than a week.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Journal only shows the current boot.</strong> Volatile storage. <code>journalctl --list-boots</code> confirms it, and the fix is the persistent-storage sequence above.</li>

<li><strong>Events in journalctl but not in /var/log/messages.</strong> Either rsyslog is not reading from the journal at all, or the <code>imjournal</code> rate limiter is discarding. Check which input module is loaded first.</li>

<li><strong>Every line appears twice in the text logs.</strong> Both <code>imjournal</code> and <code>imuxsock</code> are active while <code>ForwardToSyslog=yes</code>. Disable one path.</li>

<li><strong>Bursts of messages disappear during incidents.</strong> That is a rate limiter, and incidents are exactly when services get loud. Search for suppression notices and raise the limit for that specific unit.</li>

<li><strong>journalctl reports corruption or behaves oddly.</strong> Run <code>journalctl --verify</code>. If files are damaged, <code>journalctl --rotate</code> starts a fresh active file so new writes are clean, then vacuum the bad archives.</li>

<li><strong>&#8220;Journal has been rotated since unit was started.&#8221;</strong> Not an error. Rotation happened mid-session, so <code>journalctl -u</code> cannot map the full range. Re-run the query without the unit filter or widen the time range.</li>

<li><strong>Disk still full after vacuuming.</strong> Vacuum skips the active journal file. Rotate first, then vacuum again.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Assuming the journal is persistent because <code>journalctl</code> returns results. It always returns results. The question is how far back.</li>

<li>Disabling rsyslog on a box where fail2ban, logwatch or a scraper still reads <code>/var/log/secure</code>. Nothing errors. Detection just stops.</li>

<li>Setting <code>RateLimitBurst=0</code> globally to &#8220;stop losing logs&#8221;, then having a crash-looping service fill the disk in an afternoon.</li>

<li>Editing <code>/etc/systemd/journald.conf</code> directly and being surprised when a vendor drop-in overrides it. Use a drop-in of your own with a name that sorts later.</li>

<li>Forwarding over UDP because it is the one-line version. It drops silently under exactly the load that produced the logs you wanted.</li>

<li>Enabling a disk-assisted queue without <code>queue.maxDiskSpace</code>, turning a remote outage into a local disk-full outage.</li>

<li>Tuning journald limits on a machine whose real problem is that one application logs every health check at info level. Fix the source first.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Make storage explicit. <code>Storage=persistent</code> or <code>Storage=volatile</code>, never left to <code>auto</code> on a server you care about.</li>

<li>Always set <code>SystemKeepFree</code> alongside <code>SystemMaxUse</code>. The first bounds the journal, the second protects everything else on the volume from it.</li>

<li>Decide consciously which store is authoritative, and write it in the runbook. Two stores with no stated owner means neither gets maintained.</li>

<li>Use exactly one journal-to-rsyslog path. Both is duplication; neither is silent data loss.</li>

<li>Override rate limits per unit, not globally.</li>

<li>Monitor <code>journalctl --disk-usage</code> as a metric, not as something you check after the alert.</li>

<li>Ship off-box with TCP and a bounded disk-assisted queue, and test it with the destination firewalled off before you rely on it.</li>

<li>Keep the local copy short and the remote copy long. Local disk is the expensive place to store history.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I just disable rsyslog and use journald only?</h3>



<p class="wp-block-paragraph">On a self-contained box with no remote logging and nothing parsing text files, yes, and it saves real disk. Before you do it, grep your configuration management and cron jobs for <code>/var/log/</code> paths, and check whether fail2ban or any monitoring agent reads those files. That is where this bites people.</p>



<h3 class="wp-block-heading">Can I disable journald and use rsyslog only?</h3>



<p class="wp-block-paragraph">Not meaningfully. journald is how systemd captures unit stdout and stderr, so it stays in the path regardless. What you can do is set <code>Storage=volatile</code> with a small <code>RuntimeMaxUse</code> so it holds a short in-memory window and rsyslog owns everything durable.</p>



<h3 class="wp-block-heading">Where are journald logs stored?</h3>



<p class="wp-block-paragraph"><code>/var/log/journal/</code> when persistent, <code>/run/log/journal/</code> when volatile. They are indexed binary files, not text, so <code>grep</code> and <code>tail</code> do not work on them directly. Use <code>journalctl</code>, or <code>journalctl -o json</code> if you want to pipe structured output into something else.</p>



<h3 class="wp-block-heading">Why does journalctl show entries that never reached /var/log/messages?</h3>



<p class="wp-block-paragraph">Three usual causes: rsyslog is not reading the journal at all, the <code>imjournal</code> rate limiter dropped them, or a severity filter such as <code>MaxLevelSyslog</code> or an rsyslog rule excluded that facility or priority. Check in that order.</p>



<h3 class="wp-block-heading">Is the binary journal format a problem for log analysis?</h3>



<p class="wp-block-paragraph">Locally, no. Structured fields and fast filtering are genuinely better than parsing text with regex. It becomes a problem when you need to move logs somewhere else, because almost every collector expects text or JSON. That conversion step is precisely the role rsyslog or a modern shipper plays.</p>



<h3 class="wp-block-heading">How much disk should the journal be allowed to use?</h3>



<p class="wp-block-paragraph">Work backwards from how far back you actually query locally. If you rarely look past yesterday, a few hundred megabytes with <code>MaxRetentionSec</code> set to two or three days is plenty. Anything older belongs in a remote store where it is cheaper and survives the machine.</p>



<h3 class="wp-block-heading">Does journald compress logs automatically?</h3>



<p class="wp-block-paragraph">Yes. <code>Compress=</code> defaults to enabled and applies to entries above a size threshold, which is one reason the journal is often smaller than the equivalent text files despite carrying more metadata. It does not remove the need for a size cap.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The journald vs rsyslog question is not a contest. On a stock server both are running, both are storing the same events, and the defaults were chosen by your distribution rather than by you. The failure modes all come from that: a journal in RAM that vanishes on reboot, two uncoordinated retention policies eating the same disk, and two rate limiters dropping messages without anyone noticing.</p>



<p class="wp-block-paragraph">Run <code>systemd-analyze cat-config systemd/journald.conf</code> and <code>journalctl --list-boots</code> on your servers. Two commands, thirty seconds, and you will know whether the logs you would reach for during an incident are actually there.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help sorting out your logging pipeline?</h2>



<p class="wp-block-paragraph">Logging is one of those areas where the defaults work well enough to hide the problem until an incident makes it expensive. If any of the above sounded familiar, these are the things I take on:</p>



<ul class="wp-block-list">
<li>Auditing journald and rsyslog on existing servers and telling you exactly where each log line lives, how long it survives, and what is being silently dropped.</li>

<li>Fixing volatile journals, runaway <code>/var</code> growth and duplicate text-and-binary storage on small VPS instances.</li>

<li>Setting up reliable off-box forwarding with disk-assisted queues, TCP or RELP transport, and TLS where it is required.</li>

<li>Migrating from text-file scraping to a structured pipeline into Loki, Elasticsearch or a hosted log platform, without losing the alerts you already depend on.</li>

<li>Tuning rate limits and retention per service so noisy applications stop hiding the messages that matter.</li>

<li>Writing the runbook that says which store is authoritative, so the next engineer does not have to reverse-engineer it during an outage.</li>
</ul>



<p class="wp-block-paragraph">Send me your <code>journald.conf</code>, your <code>rsyslog.conf</code>, or the output of <code>journalctl --disk-usage</code> and I will tell you what I see before we talk about scope.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/journald-vs-rsyslog-where-logs-live/">journald vs rsyslog: Where Your Linux Logs Actually Live</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>HIPAA Compliance on AWS: The Gaps That Pass Every Security Check</title>
		<link>https://john-nessime.com/blog/technical-guides/hipaa-compliance-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 25 Jul 2026 12:04:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Config]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[Healthcare Cloud]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[Restore Testing]]></category>
		<category><![CDATA[VPC]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=184</guid>

					<description><![CDATA[<p>A working engineer's guide to HIPAA compliance on AWS, organised by the gap between the control you configured and the obligation you actually carry. Covers BAA account scope, the eligible services list as a contract boundary, KMS key policy versus the encryption checkbox, what "six years" really applies to, backup and restore scope, and the subprocessor chain nobody inventories.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/hipaa-compliance-aws/">HIPAA Compliance on AWS: The Gaps That Pass Every Security Check</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket usually reads something like: &#8220;Legal wants to know if the analytics account is in scope.&#8221; So you open Security Hub. Green. You check the Config rules. Passing. Every bucket is encrypted, every volume is encrypted, MFA is on, CloudTrail is running in all Regions. You reply that the account is fine.</p>



<p class="wp-block-paragraph">Then someone points out that a nightly job has been copying a de-identified extract into that account for eight months, the de-identification script never removed the admission dates, and the account was spun up before anyone thought about the Business Associate Addendum. Nothing was misconfigured. Every control you built worked exactly as designed. And you have been out of compliance the entire time.</p>



<p class="wp-block-paragraph">That is the shape of most real failures here. Not a breach, not a misconfiguration, but a mismatch between the boundary your tooling checks and the boundary your obligation actually follows. This post covers HIPAA compliance on AWS organised by those gaps: where the contract stops, where encryption stops being a control, what &#8220;six years&#8221; genuinely applies to, and which parts of the estate people forget are in scope at all.</p>



<h2 class="wp-block-heading">Eligible is not compliant, and the difference is the whole job</h2>



<p class="wp-block-paragraph">AWS does not sell HIPAA compliance. It sells HIPAA <em>eligible</em> services, which is a genuinely different thing. Eligible means AWS has built the service so it can lawfully handle electronic protected health information and has agreed to cover it under a Business Associate Addendum. Compliant describes an entire system: your architecture, your key management, your access reviews, your policies, your staff, your vendors.</p>



<p class="wp-block-paragraph">Under the shared responsibility model, AWS secures the infrastructure. You secure everything you build on it. Nothing about signing the BAA transfers a single obligation off your side of the line. An unencrypted RDS instance, an overly broad IAM policy or an application that logs a patient identifier into stdout is your problem in exactly the same way it would be in a rack you own.</p>



<p class="wp-block-paragraph">People know this in the abstract. Where it bites is in the specifics below.</p>



<h2 class="wp-block-heading">Gap one: the BAA is a contract boundary, and nothing enforces it</h2>



<p class="wp-block-paragraph">This is the one I would fix first, because it is invisible to every security tool you own.</p>



<p class="wp-block-paragraph">The AWS BAA is self-service through AWS Artifact, at no extra cost. You can accept it for a single account, or, if you are in the management account of an AWS Organization, accept it once so that existing and future member accounts are covered. That organization-level option is the one worth using, because the per-account version quietly rots: someone creates a new account for a proof of concept, nobody repeats the Artifact step, and six months later that account is running something real.</p>



<p class="wp-block-paragraph">The second half of the boundary is the HIPAA Eligible Services Reference that AWS publishes. Only services on that list may create, receive, process, maintain or transmit ePHI under the BAA. The list is long, it changes, and some entries carry carve-outs where the service is eligible but a specific feature is not. Reading a service name on the list and assuming every feature inside it is covered is the kind of mistake that only surfaces during an audit.</p>



<p class="wp-block-paragraph">Here is the part worth internalising: <strong>there is no AWS control that stops you putting PHI into a non-eligible service.</strong> No API error, no Config rule out of the box, no GuardDuty finding. The eligible services list is a contractual construct. Your infrastructure has no idea it exists.</p>



<h3 class="wp-block-heading">Turning a contract boundary into a technical one</h3>



<p class="wp-block-paragraph">The mechanism that actually helps is Service Control Policies on the organizational unit that holds your PHI accounts. SCPs set the ceiling on what any principal in those accounts can do, including the root user, so they work as a guardrail rather than a suggestion.</p>



<p class="wp-block-paragraph">Start with the easy one. Pin the accounts to the Regions you have actually assessed, because data residency assumptions fall apart the moment someone launches something in a Region you never reviewed:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnapprovedRegions",
      "Effect": "Deny",
      "NotAction": [
        "iam:*",
        "organizations:*",
        "route53:*",
        "cloudfront:*",
        "support:*",
        "sts:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The <code>NotAction</code> list matters. Global services are backed by endpoints in specific Regions, so denying them wholesale by Region locks you out of IAM and breaks Route 53 and CloudFront. Those entries are exemptions, not an allow-list.</p>



<p class="wp-block-paragraph">The harder one is restricting which services can be used at all. The same <code>NotAction</code> pattern works, with the services you have approved for PHI listed as the exemptions and everything else denied. It is effective and it is blunt: every new service anyone wants becomes a change request against the policy, and if you forget a dependency you find out through a failure in production. I would only reach for it on a dedicated PHI OU where the workload is well understood, not across a general-purpose organization.</p>



<p class="wp-block-paragraph">Whichever route you take, write down the approved service list somewhere a human reviews on a schedule, and diff it against the AWS reference periodically. That review is itself a compliance artefact.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap two: encryption is a checkbox, the key policy is the control</h2>



<p class="wp-block-paragraph">Almost every guide to HIPAA compliance on AWS tells you to encrypt at rest and in transit. Almost none of them explain why it is worth doing properly rather than minimally, so teams enable default encryption with an AWS-managed key, watch the Config rule turn green, and move on.</p>



<p class="wp-block-paragraph">The reason to care is the Breach Notification Rule. It applies to <em>unsecured</em> PHI, meaning PHI that has not been rendered unusable, unreadable or indecipherable through a method HHS has specified. HHS guidance points at NIST-validated encryption. If PHI is encrypted to that standard and the decryption keys were not compromised alongside it, an incident involving that data generally does not trigger the notification machinery at all. No individual letters, no HHS portal submission, no press release for a large incident.</p>



<p class="wp-block-paragraph">Read that second condition again, because it is where the architecture decision lives. The safe harbour depends on the keys not being compromised with the data. If your encryption key is one an attacker inherits automatically the moment they compromise a role in the account, you have encryption but you may not have the argument.</p>



<h3 class="wp-block-heading">What that means in practice</h3>



<ul class="wp-block-list">
<li>Use customer managed KMS keys for anything holding PHI, not AWS-managed keys. Only a customer managed key gives you a key policy you can write, and only a key policy lets you deny decryption independently of the resource policy.</li>

<li>Separate the key administrators from the key users. The people who can schedule deletion of a key should not be the people whose application role uses it every second.</li>

<li>Use a distinct key per data domain rather than one key for the whole account. Blast radius and audit trail both improve, and you get the ability to revoke access to one dataset without touching another.</li>

<li>Constrain key usage with the <code>kms:ViaService</code> condition so a key that exists to encrypt RDS storage cannot be used to decrypt something a role dragged into Lambda.</li>

<li>Turn on key rotation and leave it on. It costs nothing operationally and it is the kind of thing an assessor asks about by reflex.</li>
</ul>



<p class="wp-block-paragraph">Pull the current key policy before you assume it says what you think:</p>



<pre class="wp-block-code"><code>aws kms get-key-policy 
  --key-id alias/phi-rds 
  --policy-name default 
  --output text

# Find storage that slipped through unencrypted
aws ec2 describe-volumes 
  --filters Name=encrypted,Values=false 
  --query 'Volumes[].{Id:VolumeId,AZ:AvailabilityZone}' 
  --output table

aws rds describe-db-instances 
  --query 'DBInstances[?StorageEncrypted==`false`].DBInstanceIdentifier' 
  --output text</code></pre>



<p class="wp-block-paragraph">The RDS query is the important one, because RDS encryption cannot be enabled in place. If that command returns anything, the fix is a snapshot, an encrypted copy of the snapshot, a restore, and a cutover. Plan for downtime or a replication strategy. This is the single most common &#8220;we will fix it later&#8221; item I see, and later gets expensive.</p>



<p class="wp-block-paragraph">Also switch on EBS encryption by default in every Region you use, so the next instance somebody launches from a console wizard is not a new exception:</p>



<pre class="wp-block-code"><code>aws ec2 enable-ebs-encryption-by-default --region us-east-1
aws ec2 get-ebs-encryption-by-default --region us-east-1</code></pre>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap three: you have logs, but you may not have evidence</h2>



<p class="wp-block-paragraph">The Security Rule requires audit controls: mechanisms that record and examine activity in systems containing ePHI. It also requires you to regularly review records of information system activity. Both of those are about having and using the records.</p>



<p class="wp-block-paragraph">Now the correction, because this one is repeated everywhere and it is wrong in a way that costs money. You will read that HIPAA requires six years of audit logs. It does not. The six-year requirement sits in the documentation standard, and it applies to the policies, procedures and records of actions, activities and assessments that the Security Rule requires you to keep, retained for six years from creation or from when the document was last in effect, whichever is later. There is no clause anywhere in the Security Rule that names a retention period for CloudTrail events.</p>



<p class="wp-block-paragraph">What this actually means is more demanding, not less. You have to <em>decide</em> your log retention period, write it into a policy, justify it against your risk analysis, and then keep that policy for six years. And an assessor will hold you to the number you wrote. Setting a CloudWatch Logs retention of thirty days while your policy claims one year is a finding. Storing seven years of everything because a blog told you to, when your policy says two, is not compliance, it is just a bill.</p>



<p class="wp-block-paragraph">So: pick a period you can defend, make the infrastructure match it exactly, and treat any gap between policy and configuration as a defect.</p>



<h3 class="wp-block-heading">Making logs into evidence</h3>



<p class="wp-block-paragraph">Retention is only half of it. The other half is being able to show that the records were not altered. CloudTrail has log file validation for exactly this, and it is off unless you turn it on:</p>



<pre class="wp-block-code"><code>aws cloudtrail update-trail 
  --name org-phi-trail 
  --enable-log-file-validation

# Later, prove a window of logs is intact
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:us-east-1:111122223333:trail/org-phi-trail 
  --start-time "$(date -u -d '90 days ago' +%Y-%m-%dT%H:%M:%SZ)"</code></pre>



<p class="wp-block-paragraph">With validation enabled, CloudTrail writes signed digest files alongside the log files, and <code>validate-logs</code> checks them. The difference between &#8220;here are our logs&#8221; and &#8220;here are our logs, and here is a cryptographic check that nothing was modified or deleted&#8221; is the difference between an assertion and evidence.</p>



<p class="wp-block-paragraph">Put the archive bucket in a separate account that the workload accounts cannot write to or delete from, and apply S3 Object Lock in compliance mode for the retention window you committed to. Object Lock in compliance mode cannot be shortened or bypassed by anyone, including the root user, which is exactly the property you want and exactly the property that will hurt if you set the period carelessly. Test it in governance mode first.</p>



<p class="wp-block-paragraph">For the review obligation, a query interface matters more than raw storage. Athena over the CloudTrail bucket is the cheap default. If you want alerting and dashboards on top of access patterns, this is a natural place for a platform such as Grafana, Datadog or Splunk, and any of them will hold access records for you. Just remember that if those records contain PHI, that vendor needs a BAA too. See the subprocessor section below.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap four: backups, snapshots and the parts of scope people forget</h2>



<p class="wp-block-paragraph">The Security Rule&#8217;s contingency plan standard is not optional decoration. It requires a data backup plan, a disaster recovery plan and an emergency mode operation plan, plus testing and revision procedures. Most teams have the backups. Far fewer have the tested restore, and the tested restore is the part that gets asked about.</p>



<p class="wp-block-paragraph">Three things routinely go wrong here.</p>



<ol class="wp-block-list">
<li><strong>Copies leave the boundary.</strong> A cross-Region snapshot copy lands in a Region you did not assess. A cross-account copy for the DR account lands somewhere outside the OU your SCPs protect. The data is still PHI. The controls did not travel with it.</li>

<li><strong>Re-encryption changes the key, not just the copy.</strong> Copying an encrypted snapshot to another account requires a key the destination can use. It is easy to end up with a shared or less restrictive key protecting your backups than protects production, which inverts the risk model.</li>

<li><strong>The restore is never rehearsed.</strong> A backup you have never restored is a hypothesis. Schedule a restore into an isolated account, record the elapsed time, and file the result. That record is your evidence for the testing requirement, and it is the single easiest compliance artefact to produce for free.</li>
</ol>



<p class="wp-block-paragraph">While you are inventorying, remember the places PHI ends up without anyone deciding it should: application logs that include request bodies, database slow query logs capturing parameter values, support tickets with screenshots attached, CSV extracts in an analyst&#8217;s bucket, and non-production environments seeded from a production dump. That last one is the classic. If your staging database is a copy of production, staging is in scope, and staging is almost never built to the same standard.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap five: the business associate chain does not stop at AWS</h2>



<p class="wp-block-paragraph">Your BAA with AWS covers AWS. It covers nothing else in your stack.</p>



<p class="wp-block-paragraph">Every vendor that can create, receive, maintain or transmit PHI on your behalf is a business associate and needs an agreement. In a typical AWS estate that means the error tracker holding stack traces, the log aggregation platform, the APM tool, the transactional email provider, the customer support desk, the CI system if it ever touches a production dataset, and any AI or analytics service you have wired in.</p>



<p class="wp-block-paragraph">Build the inventory as a table with three columns: vendor, what PHI it can see, and whether a signed agreement exists. The third column is usually where the surprises are. Some vendors sign readily, some only on higher-priced tiers, and some decline entirely, at which point you have an architecture decision rather than a procurement one.</p>



<p class="wp-block-paragraph">One structural move that reduces this surface considerably: keep everything that does not need PHI out of the PHI accounts entirely. Your marketing site, your docs, your status page and your public API gateway for non-clinical traffic do not belong in a regulated account. Running them on ordinary infrastructure, whether that is a separate AWS account, a straightforward VPS from a host like InterServer, or a static site behind Cloudflare, shrinks the estate you have to assess, evidence and defend. Fewer things in scope is the cheapest compliance win available.</p>



<p class="wp-block-paragraph">For tracking the paperwork side, compliance automation platforms such as Vanta, Drata or Secureframe pull evidence from AWS on a schedule and keep the vendor register current. They are genuinely useful for the collection and reminder burden. They do not design your architecture, and I have seen teams treat a green dashboard in one of those tools as though it were an assessment. It is not. It is a checklist that knows what you told it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What is changing, and why &#8220;addressable&#8221; is a bad thing to build on</h2>



<p class="wp-block-paragraph">Since it was adopted, the Security Rule has split implementation specifications into <em>required</em> and <em>addressable</em>. Addressable never meant optional. It meant you assess whether the specification is reasonable and appropriate, and if not, you implement an equivalent alternative or document why neither is necessary. In practice, a lot of organisations turned the documented justification into the deliverable and skipped the control.</p>



<p class="wp-block-paragraph">HHS published a Notice of Proposed Rulemaking in the Federal Register in January 2025 that would remove that distinction, making implementation specifications required with limited exceptions, and would explicitly require encryption of ePHI at rest and in transit and multi-factor authentication, again with limited exceptions. The comment period closed in March 2025.</p>



<p class="wp-block-paragraph">Be precise about the status, because a lot of vendor content is not: <strong>this is a proposed rule and it is not final.</strong> The expected timeline for final action has slipped more than once, and the requirements could still change or be withdrawn. Nobody should be telling you a compliance deadline as though it were settled.</p>



<p class="wp-block-paragraph">What is worth taking from it is the direction of travel. If your current position depends on having documented that encryption or MFA was not reasonable and appropriate, that position is fragile regardless of what the final rule says. On AWS specifically, encryption at rest and MFA are both cheap and both already best practice. Building the architecture on an addressable deferral is an unforced risk.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the findings you will actually hit</h2>



<h3 class="wp-block-heading">&#8220;An assessor asked which accounts are in BAA scope and nobody could answer&#8221;</h3>



<p class="wp-block-paragraph">Sign in to AWS Artifact from the management account and check the organization agreements tab to see whether the BAA was accepted at the organization level or per account. If it is per account, list your accounts, work out which hold PHI, and confirm each one individually. Then move to the organization-level agreement so this question has one answer forever.</p>



<h3 class="wp-block-heading">&#8220;Config says the bucket is encrypted but we cannot prove who read the objects&#8221;</h3>



<p class="wp-block-paragraph">Bucket encryption and object-level access logging are unrelated. CloudTrail management events do not record S3 object reads by default. You need CloudTrail data events for that bucket, or S3 server access logging, and both cost money proportional to request volume. Enable data events selectively on the buckets that hold PHI rather than account-wide.</p>



<h3 class="wp-block-heading">&#8220;We enabled an SCP and production broke&#8221;</h3>



<p class="wp-block-paragraph">Almost always a Region deny catching a global service endpoint, or a service allow-list missing a dependency the workload calls indirectly. Check CloudTrail for <code>AccessDenied</code> events with an explicit deny from an SCP, and look at the service name in the event rather than the one you expected. Attach new SCPs to a test OU with a representative workload before the PHI OU.</p>



<h3 class="wp-block-heading">&#8220;Snapshot copy to the DR account fails with a KMS error&#8221;</h3>



<p class="wp-block-paragraph">The destination account cannot use the source key. The source key policy has to grant the destination principal permission to use it, and the copy has to specify a key the destination can decrypt with. Fix it by granting explicitly on a key you control, not by falling back to an AWS-managed key, which is the tempting shortcut and gives up the key policy control you needed.</p>



<h3 class="wp-block-heading">&#8220;CloudWatch Logs retention was never set&#8221;</h3>



<p class="wp-block-paragraph">New log groups default to never expiring, which is both a cost problem and a policy mismatch. Audit them with <code>aws logs describe-log-groups</code> and look for groups with no <code>retentionInDays</code> value, then set the period your policy specifies with <code>aws logs put-retention-policy</code>.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the signed BAA as the finish line rather than the prerequisite. It is the thing you need before the first byte of PHI arrives, not evidence that anything is configured correctly.</li>

<li>Assuming a service is fully eligible because its name appears on the list, without reading the feature-level carve-outs next to it.</li>

<li>Quoting &#8220;six years&#8221; as a log retention requirement, then either overspending on storage or writing a policy that contradicts the actual configuration.</li>

<li>Using AWS-managed KMS keys for PHI, which leaves no key policy to write and no independent revocation path.</li>

<li>Seeding staging or test environments from production data and then holding those environments to a lower standard.</li>

<li>Forgetting that the risk analysis is a required, recurring, documented activity, not a one-off spreadsheet from the year you launched.</li>

<li>Signing a BAA with AWS and none of the ten other vendors that can see the same data.</li>

<li>Letting a compliance automation dashboard stand in for an architecture review.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li><strong>Isolate PHI in its own accounts and its own OU.</strong> Account boundaries are the strongest isolation AWS offers, and they make the scope question answerable in one sentence.</li>

<li><strong>Accept the BAA at the organization level.</strong> It removes an ongoing manual step that fails silently.</li>

<li><strong>Customer managed keys, one per data domain, with split admin and usage roles.</strong> This is what makes the breach safe harbour argument defensible rather than theoretical.</li>

<li><strong>Write the retention period down first, configure second.</strong> Policy and infrastructure should agree exactly, in both directions.</li>

<li><strong>Ship audit logs to a separate account with Object Lock and CloudTrail validation enabled.</strong> Immutability and integrity are what turn logs into evidence.</li>

<li><strong>Keep PHI out of everything that does not need it.</strong> De-identify early, tokenise where you can, and route non-clinical traffic through infrastructure that is not in scope.</li>

<li><strong>Define everything in Terraform or OpenTofu.</strong> A reviewable, version-controlled definition of your controls is worth more to an assessor than any screenshot, and it stops drift being invisible.</li>

<li><strong>Rehearse the restore and the breach response.</strong> Both are required, both are tested by asking for the record, and both are cheap to evidence if you actually do them.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is AWS HIPAA compliant?</h3>



<p class="wp-block-paragraph">Not on its own, and the phrasing is the problem. AWS offers HIPAA-eligible services and will sign a Business Associate Addendum, which means you can build a compliant system on it. Compliance is a property of your whole environment, including configuration, policies, vendors and staff. No provider can sell it to you as a finished product.</p>



<h3 class="wp-block-heading">How do I sign a BAA with AWS?</h3>



<p class="wp-block-paragraph">Through AWS Artifact in the console. It is self-service and there is no additional charge. Accept it for an individual account under account agreements, or from the management account of an AWS Organization under organization agreements so all current and future member accounts are covered. It should be accepted by someone with authority to bind your organisation, and it must be in place before any PHI reaches AWS.</p>



<h3 class="wp-block-heading">Does HIPAA require six years of CloudTrail logs?</h3>



<p class="wp-block-paragraph">No. The six-year requirement is a documentation retention rule covering the policies, procedures and records the Security Rule requires you to maintain, kept for six years from creation or from when they were last in effect. The audit controls standard requires the mechanism to record and examine activity but does not name a retention period for the logs themselves. You set that period in your own policy, justify it, and make the configuration match.</p>



<h3 class="wp-block-heading">Which AWS services can I use with PHI?</h3>



<p class="wp-block-paragraph">Only those on the AWS HIPAA Eligible Services Reference, and only in accounts covered by your BAA. Check the list before adopting anything new, read the feature-level exclusions noted against individual services, and re-check periodically because entries are added over time. Nothing in AWS will stop you using a non-eligible service with PHI, so this has to be an explicit process on your side.</p>



<h3 class="wp-block-heading">If encrypted PHI is exposed, do I still have to report a breach?</h3>



<p class="wp-block-paragraph">Generally no, provided the encryption meets the standard in HHS guidance and the decryption keys were not compromised along with the data. The Breach Notification Rule applies to unsecured PHI, and properly encrypted data does not meet that definition. This is why key management, not just enabling encryption, is the part that determines whether the protection is real. You still document the incident and the assessment.</p>



<h3 class="wp-block-heading">Are the new HIPAA Security Rule requirements in force?</h3>



<p class="wp-block-paragraph">Not at the time of writing. The proposals to make all implementation specifications required and to mandate encryption and multi-factor authentication came from a Notice of Proposed Rulemaking published in January 2025. The comment period has closed, but no final rule has been issued and the timeline has moved. Treat any specific compliance deadline you see quoted with suspicion and check the current status directly.</p>



<h3 class="wp-block-heading">Is a HIPAA-compliant AWS environment expensive to run?</h3>



<p class="wp-block-paragraph">The controls themselves are mostly cheap. KMS keys, CloudTrail validation, Config rules and account separation cost very little. The real costs are log storage volume, CloudTrail data events on busy buckets, running non-production environments to production standard, and staff time on risk analysis and evidence collection. Reducing what is in scope is the most effective cost lever, which is another reason to keep non-clinical workloads out of the regulated accounts.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">HIPAA compliance on AWS fails at the seams, not at the controls. Your encryption works. Your IAM policies are tight. What goes wrong is that the obligation follows the data into an account nobody added to the BAA, a Region nobody assessed, a staging database seeded from production, a vendor nobody signed an agreement with, or a retention period nobody wrote down.</p>



<p class="wp-block-paragraph">So build the boundary technically rather than trusting it contractually. Isolate PHI into its own accounts, wrap those accounts in guardrails that make the contract boundary enforceable, own your keys so the encryption means something legally, and make your logs provable rather than merely present. Then write the whole thing down, because in this domain the documentation genuinely is part of the control.</p>



<p class="wp-block-paragraph">None of that is exotic engineering. It is ordinary AWS work applied to a boundary that no dashboard draws for you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Working on a healthcare workload on AWS?</h2>



<p class="wp-block-paragraph">This is the kind of work I do. If you are building or inheriting a PHI environment on AWS, I can help with:</p>



<ul class="wp-block-list">
<li><strong>Scope and boundary review:</strong> mapping which accounts, Regions, services and vendors actually touch PHI, and finding the ones nobody knew about.</li>

<li><strong>Account and OU design with enforceable guardrails:</strong> SCPs, organization-level BAA coverage, and Region and service restrictions that hold without breaking your workloads.</li>

<li><strong>KMS key architecture:</strong> customer managed keys per data domain, split administration and usage, and key policies written so the breach safe harbour argument stands up.</li>

<li><strong>Audit logging that produces evidence:</strong> centralised CloudTrail with log file validation, an isolated archive account with Object Lock, and retention that matches your written policy exactly.</li>

<li><strong>Backup, restore and contingency testing:</strong> encrypted cross-account copies that stay inside your boundary, plus rehearsed restores documented as compliance artefacts.</li>

<li><strong>Terraform or OpenTofu modules for the whole control set,</strong> so your posture is reviewable, repeatable and does not drift between audits.</li>
</ul>



<p class="wp-block-paragraph">If you would rather start with something concrete than a discovery call, send me a redacted account structure, an SCP that is causing trouble, or the output of a Config or Security Hub run, and I will tell you what I would look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/hipaa-compliance-aws/">HIPAA Compliance on AWS: The Gaps That Pass Every Security Check</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
