<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Serve-Stale | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/serve-stale/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/serve-stale/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Mon, 09 Nov 2026 07:00:41 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Serve-Stale | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/serve-stale/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>DNS Cutover: TTL Planning and the Rollback Window Nobody Sizes Properly</title>
		<link>https://john-nessime.com/blog/technical-guides/dns-cutover-ttl-rollback-window/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 09 Nov 2026 07:00:41 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Networking]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Connection Pooling]]></category>
		<category><![CDATA[Cutover Planning]]></category>
		<category><![CDATA[dig]]></category>
		<category><![CDATA[DNS]]></category>
		<category><![CDATA[DNS Cache]]></category>
		<category><![CDATA[DNS Migration]]></category>
		<category><![CDATA[DNSSEC]]></category>
		<category><![CDATA[Nameservers]]></category>
		<category><![CDATA[Negative Caching]]></category>
		<category><![CDATA[Reliability Engineering]]></category>
		<category><![CDATA[Rollback Plan]]></category>
		<category><![CDATA[Serve-Stale]]></category>
		<category><![CDATA[SERVFAIL]]></category>
		<category><![CDATA[TTL]]></category>
		<category><![CDATA[Unbound]]></category>
		<category><![CDATA[Website Migration]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=371</guid>

					<description><![CDATA[<p>Lowering your TTL 48 hours ahead is the easy part of a DNS cutover. The hard part is the cache layers that never honored your TTL at all, and the rollback window that closes the moment the new origin takes its first write. Here is how to plan both, and what to do with the old origin the second you flip.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/dns-cutover-ttl-rollback-window/">DNS Cutover: TTL Planning and the Rollback Window Nobody Sizes Properly</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The migration looked clean. Records flipped at 02:00, the new origin took traffic within a few minutes, synthetic checks went green, everyone went to bed. Two days later a support ticket arrives: a customer placed an order, got a confirmation email, and the order is not in the admin panel. It is not in the new database either. It is sitting in the old one, on a server that was supposed to be idle.</p>



<p class="wp-block-paragraph">That is the failure mode a DNS cutover produces most often, and it is invisible for exactly as long as it takes someone to notice missing data. Nothing alerted. Nothing 500&#8217;d. A slice of traffic simply kept resolving to the old IP and the old origin was happy to serve it.</p>



<p class="wp-block-paragraph">Most guidance on this topic stops at &#8220;lower your TTL to 300 seconds forty-eight hours ahead.&#8221; That advice is correct and it is nowhere near sufficient. It plans the forward move and says nothing about the two things that actually determine whether the cutover is safe: which caches ignored your TTL entirely, and how long you can still reverse the decision without making things worse.</p>



<p class="wp-block-paragraph">This post covers the TTL arithmetic, the cache layers that never asked your opinion, how to size a real rollback window, and how to make the old origin safe rather than merely reachable. Commands are included where they explain something.</p>



<h2 class="wp-block-heading">The TTL math is the easy part</h2>



<p class="wp-block-paragraph">A TTL is a promise to resolvers: you may reuse this answer for this many seconds without asking again. The countdown starts when the resolver caches the answer, not when you publish it. That single detail explains almost every mistake people make here.</p>



<p class="wp-block-paragraph">Watch the countdown directly. Query the same name twice through a caching resolver and the TTL field drops:</p>



<pre class="wp-block-code"><code>dig +noall +answer app.example.com A
# app.example.com.  287  IN  A  203.0.113.10

# a few seconds later
dig +noall +answer app.example.com A
# app.example.com.  281  IN  A  203.0.113.10</code></pre>



<p class="wp-block-paragraph">Compare that with the authoritative answer, which always shows the full configured value because there is no cache in front of it:</p>



<pre class="wp-block-code"><code>dig @ns1.example-dns.net +noall +answer app.example.com A
# app.example.com.  300  IN  A  203.0.113.10</code></pre>



<h3 class="wp-block-heading">Lowering the TTL late does nothing</h3>



<p class="wp-block-paragraph">If the record currently carries a one day TTL and you drop it to 300 at 09:00, a resolver that cached the answer at 08:00 keeps serving the old IP until roughly 08:00 the next morning. Your new short TTL only applies to lookups that happen after that cache entry expires. The lower value has to be published, cached, and honored before it means anything.</p>



<p class="wp-block-paragraph">The working rule is to lower the TTL at least twice the current TTL ahead of the change, and in practice at least a full day for anything meaningful. One TTL period gets most resolvers onto the new value; the second period covers the ones that refreshed a second before you published.</p>



<h3 class="wp-block-heading">Negative answers have their own clock</h3>



<p class="wp-block-paragraph">If you create a new hostname during the cutover and something queried it beforehand, the &#8220;does not exist&#8221; answer was cached too. Negative caching is not governed by the record TTL, because the record did not exist. It is governed by the SOA of the zone, specifically the lesser of the SOA record&#8217;s own TTL and its MINIMUM field.</p>



<pre class="wp-block-code"><code>dig +noall +answer SOA example.com
# example.com. 3600 IN SOA ns1.example-dns.net. hostmaster.example.com. (
#   1042 7200 3600 1209600 3600 )
#   the final number is MINIMUM, used as the negative caching TTL</code></pre>



<p class="wp-block-paragraph">If that value is an hour and someone probed your new endpoint the day before go-live, you will spend an hour explaining why the record &#8220;is not propagating&#8221; when it published perfectly. Create new names before anything queries them, or lower the SOA minimum alongside your record TTLs.</p>



<h2 class="wp-block-heading">The caches that never asked for your TTL</h2>



<p class="wp-block-paragraph">Here is the part that gets skipped. A TTL is advisory. Plenty of software in the path between your zone and a user&#8217;s socket either clamps it, extends it, or never consults it at all.</p>



<h3 class="wp-block-heading">Resolvers with a TTL floor</h3>



<p class="wp-block-paragraph">Recursive resolvers can be configured to refuse short TTLs to cut query volume. Unbound exposes <code>cache-min-ttl</code> and <code>cache-max-ttl</code>. BIND has <code>min-cache-ttl</code> and <code>max-cache-ttl</code>. dnsmasq accepts <code>--min-cache-ttl</code> and <code>--max-cache-ttl</code>. Set a floor of twenty minutes on a corporate forwarder and every 300 second record in the world becomes a twenty minute record for that office, whatever the zone says.</p>



<p class="wp-block-paragraph">You cannot audit this from outside. What you can do is expect it, and treat your published TTL as a floor on propagation time rather than a ceiling.</p>



<h3 class="wp-block-heading">Serve-stale</h3>



<p class="wp-block-paragraph">RFC 8767 lets a resolver keep serving an expired answer when it cannot reach the authoritative servers, so a nameserver outage does not become a resolution outage. It is a good mechanism and it is widely implemented, including in Unbound via <code>serve-expired</code> and in BIND via its stale answer settings.</p>



<p class="wp-block-paragraph">The consequence for a cutover is worth stating plainly. If your authoritative nameservers are unreachable or slow at the moment you flip, some resolvers will happily continue handing out the old IP past its expiry. That is exactly the wrong time to be changing DNS providers, rotating nameservers, and repointing records in the same maintenance window.</p>



<h3 class="wp-block-heading">Application runtimes that cache forever</h3>



<p class="wp-block-paragraph">The JVM is the classic one. Its positive lookup cache is governed by the <code>networkaddress.cache.ttl</code> security property, and when a security manager is installed the default is <code>-1</code>, meaning cache for the life of the process. A long running Java service can hold a resolved address until someone restarts it, days after the DNS change.</p>



<pre class="wp-block-code"><code># in $JAVA_HOME/conf/security/java.security  (Java 11+)
# or $JAVA_HOME/jre/lib/security/java.security (Java 8)
networkaddress.cache.ttl=60
networkaddress.cache.negative.ttl=10</code></pre>



<p class="wp-block-paragraph">There is a corresponding system property, <code>sun.net.inetaddr.ttl</code>, usable as a JVM argument when you cannot edit the security file. The security property is the supported path. Other runtimes and libraries have their own resolver caches with their own defaults, and none of them are visible in a <code>dig</code> output. If a client of yours is a service rather than a browser, ask what its resolver cache does before you plan the window.</p>



<h3 class="wp-block-heading">Connections that are already open</h3>



<p class="wp-block-paragraph">DNS resolves a name to an address once, at connection setup. An HTTP keep-alive connection, a database pool, a long-polling websocket, an SFTP session pinned by a partner&#8217;s batch job: none of these re-resolve because a TTL expired. They re-resolve when they reconnect. A connection pool configured to keep idle connections alive indefinitely will sit on the old IP until the socket dies or the process restarts.</p>



<p class="wp-block-paragraph">This is why the traffic tail after a cutover is so much longer for APIs than for websites. Browsers churn connections constantly. Machine clients do not.</p>



<h2 class="wp-block-heading">Sizing the rollback window for a DNS cutover</h2>



<p class="wp-block-paragraph">People plan the forward move and assume reversing it is the same operation backwards. It is not, because a DNS cutover is not one clock. It is two, and they run in opposite directions.</p>



<ul class="wp-block-list"><li><strong>The revert clock.</strong> How long after you republish the old value until traffic actually returns. This is roughly your TTL, plus whatever floors and stale answers add, plus reconnect intervals for pooled clients.</li><li><strong>The retention clock.</strong> How long the old origin must remain able to serve correctly, counted from the flip. This one is bounded by your slowest client, not your TTL.</li></ul>



<p class="wp-block-paragraph">The useful definition of the rollback window is the period during which reverting the record makes things better rather than worse. It opens at the flip and it closes the moment the two origins have diverged enough that sending traffic back would lose data. If the new database has taken writes for an hour and there is no reverse replication, your rollback window closed an hour ago no matter what the TTL says.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>Your rollback window is bounded by data divergence, not by DNS. TTL only decides how fast the reversal takes effect once you decide to make it.</p></blockquote>



<p class="wp-block-paragraph">Write both numbers down before the change. If the retention clock is longer than the rollback window, and it usually is, you have a period where you can neither go back nor decommission. That period needs a plan, and the plan is the next section.</p>



<h2 class="wp-block-heading">Make the old origin safe, not just reachable</h2>



<p class="wp-block-paragraph">The lost order at the top of this post happened because the old origin stayed fully functional. It accepted the request, wrote to its own database, and returned a 200. The stale resolver was the trigger, but the fault was that the old endpoint was still willing to act as an authority.</p>



<p class="wp-block-paragraph">Pick one of these deliberately, at the moment of the flip:</p>



<ol class="wp-block-list"><li><strong>Proxy forward.</strong> Turn the old origin into a reverse proxy pointing at the new one. Stragglers get correct answers and correct writes. This is the option I reach for first because it makes the retention clock harmless.</li><li><strong>Read-only.</strong> Keep serving reads, reject writes with a clear error. Good when proxying is impractical and losing a write is worse than showing an error.</li><li><strong>Hard stop.</strong> Return 503 for everything. Loud, unambiguous, and it converts a silent data problem into a visible availability problem. Sometimes that is the right trade.</li></ol>



<p class="wp-block-paragraph">What you should not do is leave it running as it was. A minimal Nginx proxy on the old host closes the gap in a few lines:</p>



<pre class="wp-block-code"><code>location / {
    proxy_pass https://new-origin.example.com;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Stale-Origin "1";
}</code></pre>



<p class="wp-block-paragraph">The custom header is the point. It marks every request that arrived via the old address, so you can count stragglers at the new origin and know when the tail has actually ended. Without a marker like that you are guessing.</p>



<h2 class="wp-block-heading">The cutover runbook</h2>



<ol class="wp-block-list"><li><strong>Inventory every name.</strong> Apex, www, API hosts, mail, staging, certificate validation names, CDN aliases, IPv6 records, and anything a partner hardcoded. Export the zone if your provider allows it.</li><li><strong>Record the current TTL of each name you will touch.</strong> The largest one sets your lead time.</li><li><strong>Lower TTLs at least two of those periods ahead.</strong> 300 seconds is a reasonable target. Going below 60 mostly buys query load, not speed.</li><li><strong>Lower the SOA minimum too</strong> if you are introducing new hostnames.</li><li><strong>Confirm the lower value has taken effect</strong> from several public resolvers before you commit to the window.</li><li><strong>Bring up the new origin and test it by IP or by a temporary hostname.</strong> Certificates, Host header routing, and redirects all fail differently when reached by address.</li><li><strong>Flip the records, and put the old origin into proxy, read-only, or hard-stop mode in the same step.</strong> Not five minutes later.</li><li><strong>Watch the marked requests at the new origin.</strong> When that counter reaches zero and stays there, the tail is done.</li><li><strong>Raise TTLs back</strong> once you are past the rollback window. Longer TTLs are better for resilience and load the rest of the year.</li><li><strong>Decommission last.</strong> Deleting the old host is the step you cannot undo.</li></ol>



<h2 class="wp-block-heading">Verifying from outside your own network</h2>



<p class="wp-block-paragraph">Checking resolution from your laptop tells you about one resolver. Check several, from different networks:</p>



<pre class="wp-block-code"><code>for r in 1.1.1.1 8.8.8.8 9.9.9.9 208.67.222.222; do
  printf '%-16s ' "$r"
  dig @"$r" +short app.example.com A
done</code></pre>



<p class="wp-block-paragraph">Follow the delegation from the root when an answer looks wrong, because it shows you which nameservers are actually being consulted:</p>



<pre class="wp-block-code"><code>dig +trace app.example.com A</code></pre>



<p class="wp-block-paragraph">Geography matters more than people expect, because resolver behavior varies by ISP and region. Running the same checks through exit points in a few different countries is a legitimate use for a commercial VPN such as NordVPN or Surfshark, and it is faster than asking colleagues in other offices to run <code>dig</code> for you. A synthetic monitoring product like Grafana Cloud or Cloudflare&#8217;s health checks gives you the same coverage on a schedule rather than on demand.</p>



<p class="wp-block-paragraph">Local caches lie too. Flush them before you trust a negative result:</p>



<pre class="wp-block-code"><code>resolvectl flush-caches      # systemd-resolved
ipconfig /flushdns           # Windows
sudo dscacheutil -flushcache # macOS</code></pre>



<h2 class="wp-block-heading">When the delegation itself moves</h2>



<p class="wp-block-paragraph">Changing nameservers is a different and much slower operation than changing an A record, and the two get confused constantly. The NS records that matter live in the TLD zone, and their TTL is set by the registry, not by you. For many TLDs that is on the order of a day or two, and no amount of lowering TTLs in your own zone changes it.</p>



<pre class="wp-block-code"><code>dig +norecurse NS example.com @a.gtld-servers.net</code></pre>



<p class="wp-block-paragraph">For that whole period, part of the world queries the old nameservers and part queries the new ones. Both sets must serve identical zone content, or users get different answers depending on which they happened to hit. Keep the old provider&#8217;s zone updated and running until the delegation TTL has fully expired, then stop.</p>



<p class="wp-block-paragraph">If the zone is signed, DNSSEC turns this from awkward into an outage. A DS record at the parent that no longer matches the keys of whoever is answering produces SERVFAIL for validating resolvers, which is a hard failure, not a stale answer. The safe sequencing is to get the DS state right at the parent and let its TTL expire before the delegation moves, or to run a properly coordinated multi-signer transfer where both operators publish each other&#8217;s keys. Do not treat DNSSEC as a detail to sort out after the nameservers change.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list"><li><strong>Authoritative shows the new value, a public resolver shows the old one.</strong> You are waiting on a cache. Check the TTL in the resolver&#8217;s answer; it tells you how many seconds are left.</li><li><strong>One office sees the old IP long past the TTL.</strong> Suspect a forwarder with a minimum cache TTL, or a local resolver serving expired data.</li><li><strong>A single service keeps hitting the old address.</strong> Runtime resolver cache or an open connection pool. Restarting the process confirms the diagnosis immediately.</li><li><strong>SERVFAIL after a nameserver change.</strong> Check DNSSEC first. A DS record at the parent that does not match the answering keys will do this reliably.</li><li><strong>The name resolves nowhere at all.</strong> Look for a cached negative answer and check the SOA minimum before assuming the record failed to publish.</li><li><strong>Traffic looks fully migrated but data is diverging.</strong> Your marker header count is the source of truth. If you did not add one, compare request counts at both origins.</li></ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Lowering the TTL an hour before the change, which achieves nothing.</li><li>Lowering the TTL on some records and forgetting the CNAME chain above them. Each hop caches independently.</li><li>Treating a nameserver change and a record change as the same risk.</li><li>Leaving the old origin fully writable after the flip.</li><li>Decommissioning on the same day, because monitoring looked clean.</li><li>Forgetting to raise the TTL back, then discovering the cost during a traffic spike or a resolver incident.</li><li>Testing only from the office network, which is often the least representative resolver you have access to.</li></ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list"><li>Rehearse on a throwaway hostname with the same TTL profile. Watching real propagation once teaches more than reading about it.</li><li>Give the new origin a permanent hostname of its own so you can test it without touching the name users depend on.</li><li>Mark traffic arriving via the old path so the tail is measurable rather than assumed.</li><li>Decide the rollback criteria in advance and write them down. &#8220;It looks fine&#8221; is not a criterion.</li><li>Keep DNS changes in version control if your provider has an API. Cloudflare, Route 53 and most managed providers do, and a diff beats a memory.</li><li>Do one risky thing per window. Records, nameservers, and DNSSEC changes each deserve their own.</li><li>Return to long TTLs once stable. Short TTLs are a migration tool, not a steady state.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">How long before a DNS cutover should I lower the TTL?</h3>



<p class="wp-block-paragraph">At least twice the current TTL, and a full day is a sensible minimum for anything that currently sits at an hour or more. The reason is mechanical: resolvers that cached the record under the old TTL have to let that entry expire before they ever see the new lower value.</p>



<h3 class="wp-block-heading">What TTL should I use during the change?</h3>



<p class="wp-block-paragraph">300 seconds covers most cases. Going lower increases query volume against your authoritative nameservers for very little additional speed, and some resolvers clamp short values upward anyway. For a high traffic apex domain, measure the query volume impact before dropping to 60.</p>



<h3 class="wp-block-heading">Why is one location still seeing the old IP after the TTL expired?</h3>



<p class="wp-block-paragraph">Almost always a cache that does not honor your TTL. A forwarding resolver with a minimum cache TTL, a resolver serving expired data because it could not reach your nameservers, an application runtime holding a resolved address, or a connection that never closed.</p>



<h3 class="wp-block-heading">How long should I keep the old server running?</h3>



<p class="wp-block-paragraph">Until traffic arriving via the old address has been zero for long enough that you trust the measurement, which for API-heavy workloads can be days rather than hours. Keeping it as a proxy to the new origin makes that wait cheap and safe.</p>



<h3 class="wp-block-heading">Does a low TTL let me roll back instantly?</h3>



<p class="wp-block-paragraph">It makes the DNS part fast. It does nothing about data written to the new system while the cutover was live. If there is no path back for that data, the rollback window closed the moment the first write landed, regardless of TTL.</p>



<h3 class="wp-block-heading">Is changing nameservers riskier than changing records?</h3>



<p class="wp-block-paragraph">Considerably. The delegation TTL is controlled by the registry, often measured in days, and both old and new nameservers answer during that period. Add DNSSEC and a mismatched DS record becomes SERVFAIL rather than a stale answer.</p>



<h3 class="wp-block-heading">Should I raise the TTL back afterwards?</h3>



<p class="wp-block-paragraph">Yes, once you are past the rollback window. Longer TTLs reduce load on your authoritative servers, shave a lookup off the critical path for users, and give resolvers something to fall back on if your DNS provider has a bad day.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you take one thing from this: a DNS cutover is not finished when the record changes, and the TTL you published is a floor on how long the old answer survives, never a ceiling. Plan the TTL arithmetic, then plan for the caches that ignore it, then decide what the old origin does the second you flip. That last decision is the one that turns a silent data loss into a non-event.</p>



<p class="wp-block-paragraph">Write down two numbers before every DNS cutover: how long until a reversal takes effect, and how long the old side must keep behaving correctly. If you cannot answer both, you do not have a rollback window. You have a hope.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your cutover plan?</h2>



<p class="wp-block-paragraph">Migrations are one of the few operations where an hour of review before the change is worth more than a day of firefighting after it. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing a zone before a move: every record that needs touching, the TTL lead time each one imposes, and the ones people usually miss.</li><li>Writing the cutover runbook itself, with explicit rollback criteria and a defined retention period for the old origin.</li><li>Setting up the old host as a proxy or read-only endpoint so stale resolvers cannot cause data divergence.</li><li>Nameserver and DNS provider migrations, including DNSSEC sequencing so the delegation change does not turn into SERVFAIL.</li><li>Instrumenting the tail: marking traffic that arrives via the old path and building the check that tells you when decommissioning is actually safe.</li><li>Post-cutover review when something did go wrong and you need to know which cache layer caused it.</li></ul>



<p class="wp-block-paragraph">Send me your current zone export, the record list you plan to change, or the <code>dig</code> output that is confusing you, and I will tell you what I would do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/dns-cutover-ttl-rollback-window/">DNS Cutover: TTL Planning and the Rollback Window Nobody Sizes Properly</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
