<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Negative Caching | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/negative-caching/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/negative-caching/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Mon, 09 Nov 2026 07:00:41 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Negative Caching | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/negative-caching/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>DNS Cutover: TTL Planning and the Rollback Window Nobody Sizes Properly</title>
		<link>https://john-nessime.com/blog/technical-guides/dns-cutover-ttl-rollback-window/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 09 Nov 2026 07:00:41 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Networking]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Connection Pooling]]></category>
		<category><![CDATA[Cutover Planning]]></category>
		<category><![CDATA[dig]]></category>
		<category><![CDATA[DNS]]></category>
		<category><![CDATA[DNS Cache]]></category>
		<category><![CDATA[DNS Migration]]></category>
		<category><![CDATA[DNSSEC]]></category>
		<category><![CDATA[Nameservers]]></category>
		<category><![CDATA[Negative Caching]]></category>
		<category><![CDATA[Reliability Engineering]]></category>
		<category><![CDATA[Rollback Plan]]></category>
		<category><![CDATA[Serve-Stale]]></category>
		<category><![CDATA[SERVFAIL]]></category>
		<category><![CDATA[TTL]]></category>
		<category><![CDATA[Unbound]]></category>
		<category><![CDATA[Website Migration]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=371</guid>

					<description><![CDATA[<p>Lowering your TTL 48 hours ahead is the easy part of a DNS cutover. The hard part is the cache layers that never honored your TTL at all, and the rollback window that closes the moment the new origin takes its first write. Here is how to plan both, and what to do with the old origin the second you flip.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/dns-cutover-ttl-rollback-window/">DNS Cutover: TTL Planning and the Rollback Window Nobody Sizes Properly</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The migration looked clean. Records flipped at 02:00, the new origin took traffic within a few minutes, synthetic checks went green, everyone went to bed. Two days later a support ticket arrives: a customer placed an order, got a confirmation email, and the order is not in the admin panel. It is not in the new database either. It is sitting in the old one, on a server that was supposed to be idle.</p>



<p class="wp-block-paragraph">That is the failure mode a DNS cutover produces most often, and it is invisible for exactly as long as it takes someone to notice missing data. Nothing alerted. Nothing 500&#8217;d. A slice of traffic simply kept resolving to the old IP and the old origin was happy to serve it.</p>



<p class="wp-block-paragraph">Most guidance on this topic stops at &#8220;lower your TTL to 300 seconds forty-eight hours ahead.&#8221; That advice is correct and it is nowhere near sufficient. It plans the forward move and says nothing about the two things that actually determine whether the cutover is safe: which caches ignored your TTL entirely, and how long you can still reverse the decision without making things worse.</p>



<p class="wp-block-paragraph">This post covers the TTL arithmetic, the cache layers that never asked your opinion, how to size a real rollback window, and how to make the old origin safe rather than merely reachable. Commands are included where they explain something.</p>



<h2 class="wp-block-heading">The TTL math is the easy part</h2>



<p class="wp-block-paragraph">A TTL is a promise to resolvers: you may reuse this answer for this many seconds without asking again. The countdown starts when the resolver caches the answer, not when you publish it. That single detail explains almost every mistake people make here.</p>



<p class="wp-block-paragraph">Watch the countdown directly. Query the same name twice through a caching resolver and the TTL field drops:</p>



<pre class="wp-block-code"><code>dig +noall +answer app.example.com A
# app.example.com.  287  IN  A  203.0.113.10

# a few seconds later
dig +noall +answer app.example.com A
# app.example.com.  281  IN  A  203.0.113.10</code></pre>



<p class="wp-block-paragraph">Compare that with the authoritative answer, which always shows the full configured value because there is no cache in front of it:</p>



<pre class="wp-block-code"><code>dig @ns1.example-dns.net +noall +answer app.example.com A
# app.example.com.  300  IN  A  203.0.113.10</code></pre>



<h3 class="wp-block-heading">Lowering the TTL late does nothing</h3>



<p class="wp-block-paragraph">If the record currently carries a one day TTL and you drop it to 300 at 09:00, a resolver that cached the answer at 08:00 keeps serving the old IP until roughly 08:00 the next morning. Your new short TTL only applies to lookups that happen after that cache entry expires. The lower value has to be published, cached, and honored before it means anything.</p>



<p class="wp-block-paragraph">The working rule is to lower the TTL at least twice the current TTL ahead of the change, and in practice at least a full day for anything meaningful. One TTL period gets most resolvers onto the new value; the second period covers the ones that refreshed a second before you published.</p>



<h3 class="wp-block-heading">Negative answers have their own clock</h3>



<p class="wp-block-paragraph">If you create a new hostname during the cutover and something queried it beforehand, the &#8220;does not exist&#8221; answer was cached too. Negative caching is not governed by the record TTL, because the record did not exist. It is governed by the SOA of the zone, specifically the lesser of the SOA record&#8217;s own TTL and its MINIMUM field.</p>



<pre class="wp-block-code"><code>dig +noall +answer SOA example.com
# example.com. 3600 IN SOA ns1.example-dns.net. hostmaster.example.com. (
#   1042 7200 3600 1209600 3600 )
#   the final number is MINIMUM, used as the negative caching TTL</code></pre>



<p class="wp-block-paragraph">If that value is an hour and someone probed your new endpoint the day before go-live, you will spend an hour explaining why the record &#8220;is not propagating&#8221; when it published perfectly. Create new names before anything queries them, or lower the SOA minimum alongside your record TTLs.</p>



<h2 class="wp-block-heading">The caches that never asked for your TTL</h2>



<p class="wp-block-paragraph">Here is the part that gets skipped. A TTL is advisory. Plenty of software in the path between your zone and a user&#8217;s socket either clamps it, extends it, or never consults it at all.</p>



<h3 class="wp-block-heading">Resolvers with a TTL floor</h3>



<p class="wp-block-paragraph">Recursive resolvers can be configured to refuse short TTLs to cut query volume. Unbound exposes <code>cache-min-ttl</code> and <code>cache-max-ttl</code>. BIND has <code>min-cache-ttl</code> and <code>max-cache-ttl</code>. dnsmasq accepts <code>--min-cache-ttl</code> and <code>--max-cache-ttl</code>. Set a floor of twenty minutes on a corporate forwarder and every 300 second record in the world becomes a twenty minute record for that office, whatever the zone says.</p>



<p class="wp-block-paragraph">You cannot audit this from outside. What you can do is expect it, and treat your published TTL as a floor on propagation time rather than a ceiling.</p>



<h3 class="wp-block-heading">Serve-stale</h3>



<p class="wp-block-paragraph">RFC 8767 lets a resolver keep serving an expired answer when it cannot reach the authoritative servers, so a nameserver outage does not become a resolution outage. It is a good mechanism and it is widely implemented, including in Unbound via <code>serve-expired</code> and in BIND via its stale answer settings.</p>



<p class="wp-block-paragraph">The consequence for a cutover is worth stating plainly. If your authoritative nameservers are unreachable or slow at the moment you flip, some resolvers will happily continue handing out the old IP past its expiry. That is exactly the wrong time to be changing DNS providers, rotating nameservers, and repointing records in the same maintenance window.</p>



<h3 class="wp-block-heading">Application runtimes that cache forever</h3>



<p class="wp-block-paragraph">The JVM is the classic one. Its positive lookup cache is governed by the <code>networkaddress.cache.ttl</code> security property, and when a security manager is installed the default is <code>-1</code>, meaning cache for the life of the process. A long running Java service can hold a resolved address until someone restarts it, days after the DNS change.</p>



<pre class="wp-block-code"><code># in $JAVA_HOME/conf/security/java.security  (Java 11+)
# or $JAVA_HOME/jre/lib/security/java.security (Java 8)
networkaddress.cache.ttl=60
networkaddress.cache.negative.ttl=10</code></pre>



<p class="wp-block-paragraph">There is a corresponding system property, <code>sun.net.inetaddr.ttl</code>, usable as a JVM argument when you cannot edit the security file. The security property is the supported path. Other runtimes and libraries have their own resolver caches with their own defaults, and none of them are visible in a <code>dig</code> output. If a client of yours is a service rather than a browser, ask what its resolver cache does before you plan the window.</p>



<h3 class="wp-block-heading">Connections that are already open</h3>



<p class="wp-block-paragraph">DNS resolves a name to an address once, at connection setup. An HTTP keep-alive connection, a database pool, a long-polling websocket, an SFTP session pinned by a partner&#8217;s batch job: none of these re-resolve because a TTL expired. They re-resolve when they reconnect. A connection pool configured to keep idle connections alive indefinitely will sit on the old IP until the socket dies or the process restarts.</p>



<p class="wp-block-paragraph">This is why the traffic tail after a cutover is so much longer for APIs than for websites. Browsers churn connections constantly. Machine clients do not.</p>



<h2 class="wp-block-heading">Sizing the rollback window for a DNS cutover</h2>



<p class="wp-block-paragraph">People plan the forward move and assume reversing it is the same operation backwards. It is not, because a DNS cutover is not one clock. It is two, and they run in opposite directions.</p>



<ul class="wp-block-list"><li><strong>The revert clock.</strong> How long after you republish the old value until traffic actually returns. This is roughly your TTL, plus whatever floors and stale answers add, plus reconnect intervals for pooled clients.</li><li><strong>The retention clock.</strong> How long the old origin must remain able to serve correctly, counted from the flip. This one is bounded by your slowest client, not your TTL.</li></ul>



<p class="wp-block-paragraph">The useful definition of the rollback window is the period during which reverting the record makes things better rather than worse. It opens at the flip and it closes the moment the two origins have diverged enough that sending traffic back would lose data. If the new database has taken writes for an hour and there is no reverse replication, your rollback window closed an hour ago no matter what the TTL says.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>Your rollback window is bounded by data divergence, not by DNS. TTL only decides how fast the reversal takes effect once you decide to make it.</p></blockquote>



<p class="wp-block-paragraph">Write both numbers down before the change. If the retention clock is longer than the rollback window, and it usually is, you have a period where you can neither go back nor decommission. That period needs a plan, and the plan is the next section.</p>



<h2 class="wp-block-heading">Make the old origin safe, not just reachable</h2>



<p class="wp-block-paragraph">The lost order at the top of this post happened because the old origin stayed fully functional. It accepted the request, wrote to its own database, and returned a 200. The stale resolver was the trigger, but the fault was that the old endpoint was still willing to act as an authority.</p>



<p class="wp-block-paragraph">Pick one of these deliberately, at the moment of the flip:</p>



<ol class="wp-block-list"><li><strong>Proxy forward.</strong> Turn the old origin into a reverse proxy pointing at the new one. Stragglers get correct answers and correct writes. This is the option I reach for first because it makes the retention clock harmless.</li><li><strong>Read-only.</strong> Keep serving reads, reject writes with a clear error. Good when proxying is impractical and losing a write is worse than showing an error.</li><li><strong>Hard stop.</strong> Return 503 for everything. Loud, unambiguous, and it converts a silent data problem into a visible availability problem. Sometimes that is the right trade.</li></ol>



<p class="wp-block-paragraph">What you should not do is leave it running as it was. A minimal Nginx proxy on the old host closes the gap in a few lines:</p>



<pre class="wp-block-code"><code>location / {
    proxy_pass https://new-origin.example.com;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Stale-Origin "1";
}</code></pre>



<p class="wp-block-paragraph">The custom header is the point. It marks every request that arrived via the old address, so you can count stragglers at the new origin and know when the tail has actually ended. Without a marker like that you are guessing.</p>



<h2 class="wp-block-heading">The cutover runbook</h2>



<ol class="wp-block-list"><li><strong>Inventory every name.</strong> Apex, www, API hosts, mail, staging, certificate validation names, CDN aliases, IPv6 records, and anything a partner hardcoded. Export the zone if your provider allows it.</li><li><strong>Record the current TTL of each name you will touch.</strong> The largest one sets your lead time.</li><li><strong>Lower TTLs at least two of those periods ahead.</strong> 300 seconds is a reasonable target. Going below 60 mostly buys query load, not speed.</li><li><strong>Lower the SOA minimum too</strong> if you are introducing new hostnames.</li><li><strong>Confirm the lower value has taken effect</strong> from several public resolvers before you commit to the window.</li><li><strong>Bring up the new origin and test it by IP or by a temporary hostname.</strong> Certificates, Host header routing, and redirects all fail differently when reached by address.</li><li><strong>Flip the records, and put the old origin into proxy, read-only, or hard-stop mode in the same step.</strong> Not five minutes later.</li><li><strong>Watch the marked requests at the new origin.</strong> When that counter reaches zero and stays there, the tail is done.</li><li><strong>Raise TTLs back</strong> once you are past the rollback window. Longer TTLs are better for resilience and load the rest of the year.</li><li><strong>Decommission last.</strong> Deleting the old host is the step you cannot undo.</li></ol>



<h2 class="wp-block-heading">Verifying from outside your own network</h2>



<p class="wp-block-paragraph">Checking resolution from your laptop tells you about one resolver. Check several, from different networks:</p>



<pre class="wp-block-code"><code>for r in 1.1.1.1 8.8.8.8 9.9.9.9 208.67.222.222; do
  printf '%-16s ' "$r"
  dig @"$r" +short app.example.com A
done</code></pre>



<p class="wp-block-paragraph">Follow the delegation from the root when an answer looks wrong, because it shows you which nameservers are actually being consulted:</p>



<pre class="wp-block-code"><code>dig +trace app.example.com A</code></pre>



<p class="wp-block-paragraph">Geography matters more than people expect, because resolver behavior varies by ISP and region. Running the same checks through exit points in a few different countries is a legitimate use for a commercial VPN such as NordVPN or Surfshark, and it is faster than asking colleagues in other offices to run <code>dig</code> for you. A synthetic monitoring product like Grafana Cloud or Cloudflare&#8217;s health checks gives you the same coverage on a schedule rather than on demand.</p>



<p class="wp-block-paragraph">Local caches lie too. Flush them before you trust a negative result:</p>



<pre class="wp-block-code"><code>resolvectl flush-caches      # systemd-resolved
ipconfig /flushdns           # Windows
sudo dscacheutil -flushcache # macOS</code></pre>



<h2 class="wp-block-heading">When the delegation itself moves</h2>



<p class="wp-block-paragraph">Changing nameservers is a different and much slower operation than changing an A record, and the two get confused constantly. The NS records that matter live in the TLD zone, and their TTL is set by the registry, not by you. For many TLDs that is on the order of a day or two, and no amount of lowering TTLs in your own zone changes it.</p>



<pre class="wp-block-code"><code>dig +norecurse NS example.com @a.gtld-servers.net</code></pre>



<p class="wp-block-paragraph">For that whole period, part of the world queries the old nameservers and part queries the new ones. Both sets must serve identical zone content, or users get different answers depending on which they happened to hit. Keep the old provider&#8217;s zone updated and running until the delegation TTL has fully expired, then stop.</p>



<p class="wp-block-paragraph">If the zone is signed, DNSSEC turns this from awkward into an outage. A DS record at the parent that no longer matches the keys of whoever is answering produces SERVFAIL for validating resolvers, which is a hard failure, not a stale answer. The safe sequencing is to get the DS state right at the parent and let its TTL expire before the delegation moves, or to run a properly coordinated multi-signer transfer where both operators publish each other&#8217;s keys. Do not treat DNSSEC as a detail to sort out after the nameservers change.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list"><li><strong>Authoritative shows the new value, a public resolver shows the old one.</strong> You are waiting on a cache. Check the TTL in the resolver&#8217;s answer; it tells you how many seconds are left.</li><li><strong>One office sees the old IP long past the TTL.</strong> Suspect a forwarder with a minimum cache TTL, or a local resolver serving expired data.</li><li><strong>A single service keeps hitting the old address.</strong> Runtime resolver cache or an open connection pool. Restarting the process confirms the diagnosis immediately.</li><li><strong>SERVFAIL after a nameserver change.</strong> Check DNSSEC first. A DS record at the parent that does not match the answering keys will do this reliably.</li><li><strong>The name resolves nowhere at all.</strong> Look for a cached negative answer and check the SOA minimum before assuming the record failed to publish.</li><li><strong>Traffic looks fully migrated but data is diverging.</strong> Your marker header count is the source of truth. If you did not add one, compare request counts at both origins.</li></ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Lowering the TTL an hour before the change, which achieves nothing.</li><li>Lowering the TTL on some records and forgetting the CNAME chain above them. Each hop caches independently.</li><li>Treating a nameserver change and a record change as the same risk.</li><li>Leaving the old origin fully writable after the flip.</li><li>Decommissioning on the same day, because monitoring looked clean.</li><li>Forgetting to raise the TTL back, then discovering the cost during a traffic spike or a resolver incident.</li><li>Testing only from the office network, which is often the least representative resolver you have access to.</li></ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list"><li>Rehearse on a throwaway hostname with the same TTL profile. Watching real propagation once teaches more than reading about it.</li><li>Give the new origin a permanent hostname of its own so you can test it without touching the name users depend on.</li><li>Mark traffic arriving via the old path so the tail is measurable rather than assumed.</li><li>Decide the rollback criteria in advance and write them down. &#8220;It looks fine&#8221; is not a criterion.</li><li>Keep DNS changes in version control if your provider has an API. Cloudflare, Route 53 and most managed providers do, and a diff beats a memory.</li><li>Do one risky thing per window. Records, nameservers, and DNSSEC changes each deserve their own.</li><li>Return to long TTLs once stable. Short TTLs are a migration tool, not a steady state.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">How long before a DNS cutover should I lower the TTL?</h3>



<p class="wp-block-paragraph">At least twice the current TTL, and a full day is a sensible minimum for anything that currently sits at an hour or more. The reason is mechanical: resolvers that cached the record under the old TTL have to let that entry expire before they ever see the new lower value.</p>



<h3 class="wp-block-heading">What TTL should I use during the change?</h3>



<p class="wp-block-paragraph">300 seconds covers most cases. Going lower increases query volume against your authoritative nameservers for very little additional speed, and some resolvers clamp short values upward anyway. For a high traffic apex domain, measure the query volume impact before dropping to 60.</p>



<h3 class="wp-block-heading">Why is one location still seeing the old IP after the TTL expired?</h3>



<p class="wp-block-paragraph">Almost always a cache that does not honor your TTL. A forwarding resolver with a minimum cache TTL, a resolver serving expired data because it could not reach your nameservers, an application runtime holding a resolved address, or a connection that never closed.</p>



<h3 class="wp-block-heading">How long should I keep the old server running?</h3>



<p class="wp-block-paragraph">Until traffic arriving via the old address has been zero for long enough that you trust the measurement, which for API-heavy workloads can be days rather than hours. Keeping it as a proxy to the new origin makes that wait cheap and safe.</p>



<h3 class="wp-block-heading">Does a low TTL let me roll back instantly?</h3>



<p class="wp-block-paragraph">It makes the DNS part fast. It does nothing about data written to the new system while the cutover was live. If there is no path back for that data, the rollback window closed the moment the first write landed, regardless of TTL.</p>



<h3 class="wp-block-heading">Is changing nameservers riskier than changing records?</h3>



<p class="wp-block-paragraph">Considerably. The delegation TTL is controlled by the registry, often measured in days, and both old and new nameservers answer during that period. Add DNSSEC and a mismatched DS record becomes SERVFAIL rather than a stale answer.</p>



<h3 class="wp-block-heading">Should I raise the TTL back afterwards?</h3>



<p class="wp-block-paragraph">Yes, once you are past the rollback window. Longer TTLs reduce load on your authoritative servers, shave a lookup off the critical path for users, and give resolvers something to fall back on if your DNS provider has a bad day.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you take one thing from this: a DNS cutover is not finished when the record changes, and the TTL you published is a floor on how long the old answer survives, never a ceiling. Plan the TTL arithmetic, then plan for the caches that ignore it, then decide what the old origin does the second you flip. That last decision is the one that turns a silent data loss into a non-event.</p>



<p class="wp-block-paragraph">Write down two numbers before every DNS cutover: how long until a reversal takes effect, and how long the old side must keep behaving correctly. If you cannot answer both, you do not have a rollback window. You have a hope.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your cutover plan?</h2>



<p class="wp-block-paragraph">Migrations are one of the few operations where an hour of review before the change is worth more than a day of firefighting after it. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing a zone before a move: every record that needs touching, the TTL lead time each one imposes, and the ones people usually miss.</li><li>Writing the cutover runbook itself, with explicit rollback criteria and a defined retention period for the old origin.</li><li>Setting up the old host as a proxy or read-only endpoint so stale resolvers cannot cause data divergence.</li><li>Nameserver and DNS provider migrations, including DNSSEC sequencing so the delegation change does not turn into SERVFAIL.</li><li>Instrumenting the tail: marking traffic that arrives via the old path and building the check that tells you when decommissioning is actually safe.</li><li>Post-cutover review when something did go wrong and you need to know which cache layer caused it.</li></ul>



<p class="wp-block-paragraph">Send me your current zone export, the record list you plan to change, or the <code>dig</code> output that is confusing you, and I will tell you what I would do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/dns-cutover-ttl-rollback-window/">DNS Cutover: TTL Planning and the Rollback Window Nobody Sizes Properly</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Serving Millions of Images From S3 Without public-read</title>
		<link>https://john-nessime.com/blog/cloud-computing/serve-s3-images-without-public-read/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 01 Nov 2026 07:06:53 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Web Performance]]></category>
		<category><![CDATA[Amazon CloudFront]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[Cache Control]]></category>
		<category><![CDATA[CDN]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Transfer Costs]]></category>
		<category><![CDATA[Image Delivery]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Negative Caching]]></category>
		<category><![CDATA[Origin Access Control]]></category>
		<category><![CDATA[Presigned URLs]]></category>
		<category><![CDATA[Signed URLs]]></category>
		<category><![CDATA[SigV4]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=729</guid>

					<description><![CDATA[<p>A public S3 object gives every viewer a second path to your bytes, one that skips your cache, ignores your headers and bills on a different rate card. Here is how to serve S3 images without public-read at scale: origin access control, the bucket policy condition that matters, the 403-versus-404 trap that turns missing thumbnails into origin load, and cache design that survives millions of objects.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/serve-s3-images-without-public-read/">Serving Millions of Images From S3 Without public-read</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Open the network tab on a page that is supposed to sit behind a CDN and read the image hostnames. If some of them come back from <code>bucket-name.s3.amazonaws.com</code> instead of your distribution, the CDN in front of those images is decoration. Those bytes are leaving S3 directly, and nothing you configure at the edge applies to them.</p>



<p class="wp-block-paragraph">The cause is almost always the same: the objects are public. Somebody set <code>public-read</code> during a migration, the CDN arrived later, and nobody closed the original door. The site still loads, so the problem stays invisible until an auditor asks who can read the bucket, or until the bill arrives shaped differently from the traffic graph.</p>



<p class="wp-block-paragraph">This post covers how to serve S3 images without public-read at real volume: the access path that replaces public objects, the error-handling trap that turns missing thumbnails into origin load, cache design when the object count runs into the millions, and what changes when some images have to be private per user.</p>



<h2 class="wp-block-heading">Public objects do not just leak, they make your CDN optional</h2>



<p class="wp-block-paragraph">The security argument against public buckets is worn out and people have stopped hearing it. The operational argument lands harder, because it costs money every month. When an object is readable by anyone there are two working paths to the same bytes, and whatever share of your traffic takes the direct one loses four things at once:</p>



<ul class="wp-block-list">
<li><strong>The cache.</strong> Direct requests never touch an edge location, so they never appear in your cache hit ratio. The dashboard says the cache is healthy because it only counts traffic that arrived.</li>

<li><strong>Your response headers.</strong> A response headers policy applies at the edge. S3 serves whatever metadata was written at upload time, so headers you added later are simply not there on the bypass path.</li>

<li><strong>Revocation.</strong> A public URL that has been scraped stays valid until you rename the key or change the bucket. There is no expiry to reach for and no signature to stop honouring.</li>

<li><strong>Predictable billing.</strong> Bytes leaving S3 for the internet bill as S3 data transfer. The same bytes leaving a CloudFront edge bill as CloudFront data transfer, and the origin leg from an AWS origin is not billed at all. Two paths, two rate cards, one invoice that no longer matches your traffic chart.</li>
</ul>



<p class="wp-block-paragraph">There is a second reason to stop reaching for that ACL: on a new bucket it will not work. S3 Object Ownership now defaults to <em>bucket owner enforced</em>, which disables ACLs entirely. A <code>PutObject</code> carrying <code>--acl public-read</code> comes back with a 400 and the error code <code>AccessControlListNotSupported</code>. Plenty of tutorials still open with that flag and now fail at step one.</p>



<h2 class="wp-block-heading">Serve S3 images without public-read using origin access control</h2>



<p class="wp-block-paragraph">Origin access control lets CloudFront, and only CloudFront, read a private bucket. CloudFront signs each origin request with SigV4 using short-lived credentials, and the bucket policy trusts the CloudFront service principal for one named distribution. The order of operations matters more than any individual step: grant the permission before you attach the OAC, or you get a window where the distribution signs requests the bucket does not yet trust and every viewer sees a 403.</p>



<ol class="wp-block-list">
<li>Confirm ACLs are off, so access is decided by policy alone rather than two overlapping systems.</li>

<li>Turn on all four Block Public Access settings at the bucket, and at the account if nothing there needs to be public.</li>

<li>Create the distribution against the bucket&#8217;s REST endpoint. Not the website endpoint.</li>

<li>Write the bucket policy trusting the CloudFront service principal, scoped to that distribution&#8217;s ARN.</li>

<li>Attach the OAC with signing behaviour set to always sign.</li>

<li>Verify from a machine with no AWS credentials that the direct URL fails and the distribution URL works.</li>
</ol>



<p class="wp-block-paragraph">Start by finding out what the bucket does today. If ownership comes back as <code>BucketOwnerEnforced</code>, ACLs are already disabled; if the second call errors because no configuration exists, that is an answer too.</p>



<pre class="wp-block-code"><code>aws s3api get-bucket-ownership-controls --bucket amzn-s3-demo-bucket
aws s3api get-public-access-block --bucket amzn-s3-demo-bucket</code></pre>



<h3 class="wp-block-heading">The bucket policy, and why the condition is the whole point</h3>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCloudFrontServicePrincipalReadOnly",
      "Effect": "Allow",
      "Principal": { "Service": "cloudfront.amazonaws.com" },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*",
      "Condition": {
        "StringEquals": {
          "AWS:SourceArn": "arn:aws:cloudfront::111122223333:distribution/EXAMPLEDIST"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Drop the <code>Condition</code> block and the policy still works, which is exactly why people drop it. What you have then is a bucket that trusts <em>every</em> CloudFront distribution in the world, including one somebody else creates this afternoon pointing at your bucket name. That is the confused deputy shape: the service is authorised, the caller behind it is never checked. The source ARN condition pins the trust to one distribution in one account.</p>



<p class="wp-block-paragraph">Before you paste that in, read it back the way an attacker would. The free <a href="https://john-nessime.com/tools/aws-iam-policy-checker/">AWS IAM policy checker</a> here takes a bucket or IAM policy and shows which wildcards grant more than they look like. No signup, and it is the fastest sanity check I know for a policy about to become load-bearing.</p>



<h3 class="wp-block-heading">Verify from outside</h3>



<pre class="wp-block-code"><code># should return 403
curl -sI https://amzn-s3-demo-bucket.s3.eu-west-1.amazonaws.com/img/hero.jpg

# should return 200
curl -sI https://d111111abcdef8.cloudfront.net/img/hero.jpg</code></pre>



<p class="wp-block-paragraph">Run both from somewhere with no AWS credentials configured. It is easy to test from your laptop, get a 200 from the S3 URL, and panic before realising you signed the request with your admin profile.</p>



<p class="wp-block-paragraph">Two constraints catch people here. OAC works only with the regular bucket endpoint, so index-document and redirect behaviour from the S3 website endpoint has to move into a CloudFront Function. And SSE-KMS objects need their own statement in the key policy letting the CloudFront service principal use the key, scoped with the same condition. A correct bucket policy plus a silent key policy produces a 403 identical to a permissions typo.</p>



<h2 class="wp-block-heading">The failure family nobody plans for: 403s that should be 404s</h2>



<p class="wp-block-paragraph">This is the one that bites at image scale, and it is invisible in a test with ten files. When a principal holds <code>s3:GetObject</code> but not <code>s3:ListBucket</code>, S3 answers a request for a key that does not exist with 403 rather than 404. That is deliberate. Without list permission, saying &#8220;not found&#8221; would confirm that other keys do exist. The standard OAC policy grants only <code>GetObject</code>, so every missing image comes back as Access Denied.</p>



<p class="wp-block-paragraph">At a few hundred images you never notice. At a few million, where thumbnail URLs come from a naming convention and some sizes were never rendered, three problems compound: your custom error response for 404 never fires because nothing returns 404, your monitoring fills with permission errors that are not permission errors, and the body S3 returns is XML, so the browser shows a broken image rather than your placeholder. Two honest options:</p>



<ul class="wp-block-list">
<li><strong>Add <code>s3:ListBucket</code> on the bucket ARN</strong> so missing keys return a real 404 and your custom error response can take over. The trade-off is that a request to the distribution root can then reach a list operation, so pair it with a function or behaviour that refuses or rewrites <code>/</code>.</li>

<li><strong>Keep the policy narrow</strong> and map 403 to your placeholder with a custom error response. Simpler, and you accept that status code alone no longer distinguishes missing from forbidden.</li>
</ul>



<p class="wp-block-paragraph">Either way, set the error caching duration deliberately. Cached for zero seconds, every request for a thumbnail that was never generated is a fresh origin fetch, and a crawler walking a broken URL pattern becomes a self-inflicted origin flood with nothing in your application logs to explain it. Negative caching is not an optimisation here. It is what stands between one bad URL template and a very odd Tuesday.</p>



<h2 class="wp-block-heading">Cache design when the object count is in the millions</h2>



<p class="wp-block-paragraph">Locking the bucket is the easy half. Making the edge absorb the traffic is where volume shows up.</p>



<h3 class="wp-block-heading">Cache-Control is written at upload time, and you will forget it</h3>



<p class="wp-block-paragraph">For an S3 origin, cache lifetime comes from object metadata, written when the object is written. There is no bulk edit later that is not a copy of every object, so put this in the upload path on day one.</p>



<pre class="wp-block-code"><code>aws s3 cp ./hero.jpg s3://amzn-s3-demo-bucket/img/hero.jpg 
  --cache-control "public, max-age=31536000, immutable"</code></pre>



<h3 class="wp-block-heading">Content-addressed keys instead of invalidations</h3>



<p class="wp-block-paragraph">If the key contains a hash of the bytes, the object never changes and <code>immutable</code> is honest. Publishing a new version writes a new key and the old one ages out on its own: no invalidation, no propagation wait, no billed paths. Invalidation exists for cases you did not design for. Treating it as a deployment step is how teams pay for thousands of paths a month and still serve stale images while it propagates.</p>



<h3 class="wp-block-heading">Keep the cache key small</h3>



<p class="wp-block-paragraph">Every dimension in the cache key multiplies the copies the edge has to store and fill. Three habits do most of the damage:</p>



<ul class="wp-block-list">
<li><strong>Forwarding all query strings.</strong> Campaign and analytics parameters do not change the bytes but they do fragment the cache. Include only parameters that select a different image, such as a width used by a resizing function.</li>

<li><strong>Putting <code>Accept</code> in the key raw.</strong> Serving WebP or AVIF by content negotiation needs <code>Accept</code> in the key, but browsers send dozens of variations of that header. Normalise it in a CloudFront Function to a small set of values, then key on the result.</li>

<li><strong>Chasing compression.</strong> JPEG, PNG, WebP and AVIF are already compressed. Brotli at the edge buys nothing on image bytes and is worth attention only for the SVG and JSON on the same distribution.</li>
</ul>



<h3 class="wp-block-heading">Origin Shield, and when it is just a surcharge</h3>



<p class="wp-block-paragraph">Origin Shield adds a designated regional caching layer that edge misses funnel through, so a cold object is fetched from the bucket once rather than once per region, billed per request on top of everything else. It pays for itself when origin misses are the problem: a long tail of images, an audience spread across continents, several distributions sharing one bucket. If the hit ratio is already high and the catalogue is small and hot, it buys nothing. Look at origin request volume before enabling it, not after.</p>



<h2 class="wp-block-heading">When some of the images are genuinely private</h2>



<p class="wp-block-paragraph">Locking the bucket makes the origin private. It does not make the URLs private, because anyone holding a distribution URL can fetch it. Per-customer images need signatures at the viewer edge as well.</p>



<ul class="wp-block-list">
<li><strong>Signed URLs</strong> suit one object at a time: a download link, an invoice scan, a single generated asset.</li>

<li><strong>Signed cookies</strong> suit a gallery or album under one path prefix, where minting a URL per image would be absurd.</li>
</ul>



<p class="wp-block-paragraph">Use trusted key groups rather than legacy account-level key pairs: they hold public keys you manage, rotating them does not need root credentials, and different groups can attach to different behaviours. Split public and private content by path prefix into separate cache behaviours, so public images do not pay for a signature check and private ones cannot be reached without one.</p>



<p class="wp-block-paragraph">Then weigh signature lifetime against page lifetime. A signature expiring in five minutes strands a page the user left open, and that failure looks like an outage to them and like nothing at all to you. Presigned S3 URLs are the other tempting answer, and they reintroduce the exact problem this post is about: they point at the bucket, bypass the edge, and bill as S3 egress.</p>



<h2 class="wp-block-heading">What this shape actually costs</h2>



<p class="wp-block-paragraph">Rates change, so work with the mechanism rather than a number you read in a blog post.</p>



<ul class="wp-block-list">
<li>Data transferred from an AWS origin such as S3 to CloudFront edge locations is not charged. A cache miss costs you an S3 request, not egress.</li>

<li>You pay for transfer out from the edge to the viewer, plus a per-request charge that varies by region and runs higher for HTTPS than HTTP.</li>

<li>At millions of small images, request charges are a real share of the bill rather than a rounding error. Object size decides whether transfer or requests dominate.</li>

<li>Cache hit ratio is the only lever that moves origin requests, origin load and edge behaviour together. Everything else is second order.</li>
</ul>



<p class="wp-block-paragraph">For a sense of the shape before committing, the <a href="https://john-nessime.com/tools/aws-cost-estimator/">AWS cost estimator</a> on this site walks through the parts of a monthly bill people forget, egress included.</p>



<p class="wp-block-paragraph">None of this assumes AWS is the right home for the images. If the origin is a VPS at a provider like Contabo or InterServer, or object storage with no egress fee such as Backblaze B2 or Cloudflare R2, that provider bills the origin leg and the arithmetic changes. The access-control principle survives the move: storage should be reachable only by the CDN, whoever provides each half.</p>



<h2 class="wp-block-heading">Troubleshooting the setup</h2>



<ul class="wp-block-list">
<li><strong>Every object returns AccessDenied through the distribution.</strong> Compare the distribution ID in the policy condition with the real one, character by character, and confirm signing is set to always sign.</li>

<li><strong>The distribution works and so does the direct S3 URL.</strong> Block Public Access is not fully on, or a statement with <code>"Principal": "*"</code> still sits underneath your new one.</li>

<li><strong>Uploads fail with &#8220;The bucket does not allow ACLs&#8221;.</strong> Something upstream still sends a canned ACL. Drop the flag or SDK parameter; with ownership enforced you do not need it.</li>

<li><strong>Only encrypted objects 403.</strong> The KMS key policy has no statement for the CloudFront service principal. A bucket policy alone is not enough for SSE-KMS.</li>

<li><strong>Only some objects 403.</strong> Check ownership on the failing keys, and remember key names are case sensitive.</li>

<li><strong>Everything works but the custom domain does not.</strong> The alternate domain name has to be on the distribution and the certificate in the region CloudFront reads certificates from. The <a href="https://john-nessime.com/tools/ssl-certificate-checker/">SSL certificate checker</a> confirms the chain and hostname coverage a browser will actually see.</li>
</ul>



<p class="wp-block-paragraph">Migrating from a legacy origin access identity? Keep both principals allowed in the bucket policy while you switch, then remove the OAI statement once the distribution finishes deploying. Removing it first is how you take an outage during a security improvement.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Locking the bucket while old public S3 URLs are still hard-coded in a database, an email template or a shipped mobile app.</li>

<li>Omitting the source ARN condition, which turns &#8220;only CloudFront&#8221; into &#8220;any CloudFront&#8221;.</li>

<li>Leaving error caching at zero, so every missing thumbnail is a fresh origin request forever.</li>

<li>Uploading without <code>Cache-Control</code> and relying on the distribution default, then wondering why a TTL change did nothing.</li>

<li>Using invalidation as a release step instead of putting a hash in the key.</li>

<li>Signing every image, including the marketing site&#8217;s, and losing cache efficiency to protect nothing.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Turn Block Public Access on at the account level, then grant exceptions deliberately rather than the other way round.</li>

<li>Keep one distribution per bucket where you can, so the source ARN condition stays a single value and a policy mistake has a small blast radius.</li>

<li>Separate public and private prefixes into different cache behaviours from the start. Retrofitting that split means changing URLs.</li>

<li>Alarm on origin request rate, not only error rate. A cache that quietly stops working looks fine on an error dashboard.</li>

<li>Re-read the bucket policy after every change. These policies accumulate statements and nobody deletes the old ones.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Can I still make an S3 bucket public if I want to?</h3>



<p class="wp-block-paragraph">On an older bucket with ACLs enabled, yes. On a new one you would have to move Object Ownership away from the enforced default and disable Block Public Access, which is several deliberate steps to arrive somewhere worse. If the images are genuinely public and there is no CDN in front, the fix is to add the CDN.</p>



<h3 class="wp-block-heading">Is origin access identity still supported?</h3>



<p class="wp-block-paragraph">It still works, and AWS documents it as legacy. OAC covers cases OAI does not, including SSE-KMS objects and newer regions. For anything new there is no reason to reach for OAI, and migrating is a bucket policy edit followed by an origin setting, in that order.</p>



<h3 class="wp-block-heading">Does OAC work with an S3 static website endpoint?</h3>



<p class="wp-block-paragraph">No. The website endpoint is treated as a custom origin and cannot be secured with OAC or OAI. Move index-document and redirect logic to a CloudFront Function against the regular bucket endpoint instead.</p>



<h3 class="wp-block-heading">How do I stop hotlinking once the bucket is private?</h3>



<p class="wp-block-paragraph">Private origin access does not address hotlinking, because the distribution URL is still open to whoever holds it. Referer checks in a CloudFront Function are easy and easy to forge. Signed URLs or cookies are the only mechanism that binds access to a session. Decide whether hotlinking is a cost problem or a rights problem first.</p>



<h3 class="wp-block-heading">Do signed URLs hurt cache hit ratio?</h3>



<p class="wp-block-paragraph">Less than people expect, because CloudFront validates the signature parameters at the edge rather than passing them to the origin as ordinary query strings. The bigger risk is the browser: mint a fresh signature on every page render and the viewer sees a new URL each time, defeating their cache even when the edge cache is fine.</p>



<h3 class="wp-block-heading">What happens to existing public URLs when I lock the bucket?</h3>



<p class="wp-block-paragraph">They start returning 403 immediately. There is no redirect available at the S3 layer, so the safe sequence is to move every reference to the distribution hostname first, watch S3 access logs until direct requests stop, then close the bucket.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">You can serve S3 images without public-read at any volume you like, and the setup is a bucket policy, an origin access control and a signing behaviour. That part is an afternoon.</p>



<p class="wp-block-paragraph">What matters is why it is worth doing beyond the audit checkbox. A public object gives every viewer a second path to your bytes, one that skips your cache, ignores your headers, cannot be revoked and bills on a different rate card. Closing it is what makes the CDN in front of your images real rather than nominal. Everything after that is tuning a system that at least has one front door.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your S3 and CloudFront setup?</h2>



<p class="wp-block-paragraph">Most of the work here is not writing the policy. It is working out what the current one allows, and who is still calling the old URLs. That is the kind of thing I do:</p>



<ul class="wp-block-list">
<li>Audit bucket policies, Block Public Access and Object Ownership across an account, and map who can genuinely read each bucket.</li>

<li>Migrate public buckets or legacy OAI distributions to origin access control without an outage window.</li>

<li>Find the paths still bypassing the CDN using access logs rather than guesswork, and close them.</li>

<li>Redesign cache keys, TTLs and error caching for image workloads where origin request volume drives the cost.</li>

<li>Set up signed URLs or signed cookies with key groups, including rotation that does not need root credentials.</li>

<li>Review SSE-KMS key policies and cross-account object ownership, and the 403s they quietly produce.</li>
</ul>



<p class="wp-block-paragraph">If you would rather start with something concrete than a call, send me your bucket policy and the output of <code>aws s3api get-public-access-block</code>, and I will tell you what it actually permits.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/aws-security-review/">Get an AWS security review</a></div>



<div class="wp-block-button is-style-outline is-style-outline--1"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/serve-s3-images-without-public-read/">Serving Millions of Images From S3 Without public-read</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
