<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Cutover Planning | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/cutover-planning/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/cutover-planning/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 08 Aug 2026 09:50:49 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Cutover Planning | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/cutover-planning/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Post-Migration Checklist: What Breaks Quietly in the First 48 Hours</title>
		<link>https://john-nessime.com/blog/devops/post-migration-checklist-48-hours/</link>
					<comments>https://john-nessime.com/blog/devops/post-migration-checklist-48-hours/#respond</comments>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 20 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Certbot]]></category>
		<category><![CDATA[Cron]]></category>
		<category><![CDATA[Cutover Planning]]></category>
		<category><![CDATA[Decommissioning]]></category>
		<category><![CDATA[DKIM]]></category>
		<category><![CDATA[DNS Migration]]></category>
		<category><![CDATA[Email Deliverability]]></category>
		<category><![CDATA[Let's Encrypt]]></category>
		<category><![CDATA[MX Records]]></category>
		<category><![CDATA[Restore Testing]]></category>
		<category><![CDATA[Rollback Plan]]></category>
		<category><![CDATA[Synthetic Monitoring]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[TTL]]></category>
		<category><![CDATA[Uptime Monitoring]]></category>
		<category><![CDATA[Website Migration]]></category>
		<category><![CDATA[WP-Cron]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=278</guid>

					<description><![CDATA[<p>Migrations rarely fail loudly. The site stays up while mail, cron, certificate renewal and the write path break silently behind it. A post-migration checklist for the first 48 hours, ordered by how long each failure can hide rather than by subsystem.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/post-migration-checklist-48-hours/">Post-Migration Checklist: What Breaks Quietly in the First 48 Hours</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket you don&#8217;t want after a migration doesn&#8217;t say the site is down. It says nobody has received a contact form email since Friday.</p>



<p class="wp-block-paragraph">The site never went down. The uptime monitor stayed green all weekend. Analytics looked normal. And a few days of enquiries went somewhere nobody thought to look.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of most migration damage. Not the 502. A 502 gets fixed inside ten minutes because everyone sees it at once, including the client. The expensive failures are the ones with no symptom on the front end: mail that authenticates on the old IP and not the new one, a cron job that never came across, a certificate that is valid today and will fail to renew long after everyone has stopped paying attention, orders written to a database that is about to be deleted.</p>



<p class="wp-block-paragraph">This post-migration checklist covers the first 48 hours after cutover. It&#8217;s ordered by when each failure actually becomes visible, not by category, because that ordering is what decides which problems you catch yourself and which ones reach you as a client ticket a fortnight later.</p>



<h2 class="wp-block-heading">Why a post-migration checklist should be ordered by detection latency</h2>



<p class="wp-block-paragraph">Most migration checklists you&#8217;ll find are grouped by subsystem: DNS, SSL, email, SEO, performance. That&#8217;s a fine way to write a document and a bad way to work a cutover, because it treats a 500 error on the homepage and a broken certificate renewal as equally urgent. They aren&#8217;t. One will be reported to you within minutes by someone who isn&#8217;t you. The other has a detection latency measured in weeks, and it lands at 3am.</p>



<p class="wp-block-paragraph">So sort your checks by how long the failure can hide. Short latency, catch it reactively. Long latency, go and look for it on purpose, while you still have the old server, the old data and a rollback path. That&#8217;s what makes 48 hours the right frame: long enough for a full daily cron cycle and for DNS to settle at any sane TTL, short enough that the old server is still sitting there with the evidence on it.</p>



<h2 class="wp-block-heading">Hour 0 to 1: prove you are actually looking at the new server</h2>



<p class="wp-block-paragraph">Almost everyone starts by loading the site in a browser and saying it looks fine. That check is worth close to nothing an hour after cutover. Your laptop may still hold the old answer, your ISP resolver definitely might, and if you added a hosts entry during testing you&#8217;re now testing the thing you already tested. Ask several resolvers directly, then bypass DNS entirely and talk to the new IP.</p>



<pre class="wp-block-code"><code># What is each public resolver answering, and how long will it keep answering it?
dig example.com A @1.1.1.1
dig example.com A @8.8.8.8
dig example.com A @9.9.9.9

# Talk to the new server directly, ignoring DNS completely
curl -sI --resolve example.com:443:203.0.113.10 https://example.com/

# Which certificate is that IP presenting for that hostname?
openssl s_client -connect 203.0.113.10:443 -servername example.com &lt;/dev/null 2&gt;/dev/null 
  | openssl x509 -noout -subject -issuer -dates</code></pre>



<p class="wp-block-paragraph">The full <code>dig</code> answer section, not <code>+short</code>, matters here because it shows the remaining TTL. That number tells you how much longer stale answers can survive, which is the single most useful figure you have in the first hour.</p>



<p class="wp-block-paragraph">The <code>--resolve</code> flag pins a hostname to an address for that one request, so you get the new server&#8217;s response with the correct SNI and Host header. It&#8217;s the closest thing to a definitive answer you&#8217;ll get while records are still in flight. If you want to see what the rest of the world sees, an exit in another country helps, and a VPN account with NordVPN or Surfshark is a cheaper way to get one than spinning up a box in a second region.</p>



<p class="wp-block-paragraph">Things to confirm in this window:</p>



<ul class="wp-block-list">
<li>The response comes from the new server, verified by something unique to it rather than by the page looking right. A temporary header or a file at a path that only exists on the new box removes all doubt. <em>Pass test: you can name the specific artefact that proves it.</em></li>

<li>The correct virtual host is selected, not a catch-all default. Sites with several domains on one server fail here quietly, serving the wrong document root over a valid certificate. <em>Pass test: every hostname you migrated returns its own content, checked individually.</em></li>

<li>The certificate chain is complete on the new box, not just the leaf. Browsers often paper over a missing intermediate using cached data; API clients and payment callbacks do not. <em>Pass test: an external SSL checker reports a complete chain from a machine that has never visited the site.</em></li>

<li>Redirect behaviour matches the old server exactly. HTTP to HTTPS, www to apex or the reverse, trailing slashes. <em>Pass test: a handful of known deep URLs return the same final status and location as before.</em></li>

<li>Any hosts file entries you added for testing are gone. Leave one in place and you&#8217;ll misread every check you run for the next two days.</li>
</ul>



<h2 class="wp-block-heading">Hour 1 to 6: the split-brain window, where you lose data instead of uptime</h2>



<p class="wp-block-paragraph">This is the part that costs real money, and it&#8217;s the part almost nobody checks, because during it both servers are healthy. Some resolvers still hand out the old address. Those visitors reach the old application, which happily accepts their form submission, their order, their uploaded file, their account signup. Then you delete that server.</p>



<p class="wp-block-paragraph">&#8220;Keep the old server running for 48 hours&#8221; is standard advice and it&#8217;s necessary, but on its own it makes the problem worse rather than better. A running old server that still accepts writes is a data-loss machine. What you want is an old server that stays reachable and stops accepting writes.</p>



<p class="wp-block-paragraph">Three options, in order of how much I&#8217;d trust them:</p>



<ol class="wp-block-list">
<li>Proxy the old server to the new one. Every request that lands on the old address gets forwarded to the new backend, so there is only ever one database. This is the only option that genuinely eliminates the split, and it&#8217;s worth the extra config on anything transactional.</li>

<li>Redirect the old server to a hostname that only resolves to the new IP. Cheap, works for GET traffic, does nothing useful for a POST that&#8217;s already in flight.</li>

<li>Put the old application into a read-only or maintenance state. Safe, but visitors on stale DNS see a broken site for the length of your TTL, so this only works if that number is genuinely small.</li>
</ol>



<p class="wp-block-paragraph">Whichever you pick, go and look at what the old box actually did after the cutover.</p>



<pre class="wp-block-code"><code># On the OLD server: has anything been written since cutover?
grep '"POST ' /var/log/nginx/access.log | tail -n 50

# Apache layouts vary; on DirectAdmin the per-domain log lives here
grep '"POST ' /var/log/httpd/domains/example.com.log | tail -n 50

# Is the old box still receiving mail for the domain?
tail -n 200 /var/log/maillog</code></pre>



<p class="wp-block-paragraph">Every POST in that log after the cutover timestamp is a potential lost record. Take the list to the application: compare row counts or highest IDs on the write-heavy tables between old and new, and reconcile before you decommission anything.</p>



<p class="wp-block-paragraph">Mail deserves its own pass. If MX records changed, messages go to whichever server the sending system had cached, and land in two different mailboxes. Keep both able to accept mail through the overlap, and check the old mailbox before you close the account, not after.</p>



<p class="wp-block-paragraph">Two more things here that won&#8217;t show up in any log you&#8217;re watching:</p>



<ul class="wp-block-list">
<li><strong>Anything pinned to an IP address.</strong> Payment gateway allowlists, webhook endpoints, database firewall rules, SMTP relay authorisations, third-party API allowlists. None of these follow a redirect. A webhook posted to the old IP returns a perfectly good 200 and the new application never hears about the payment.</li>

<li><strong>Outbound connections from the new IP.</strong> Your application is now calling other people&#8217;s APIs from an address they&#8217;ve never seen. If any of those partners allowlist by IP, the failure appears as a timeout in a background job rather than an error on a page.</li>
</ul>



<h2 class="wp-block-heading">Hour 6 to 24: everything that runs on a schedule</h2>



<p class="wp-block-paragraph">Files and databases get migrated. Scheduled work frequently does not, because it lives outside the document root and outside the database dump, and because nothing on the site breaks when it&#8217;s missing.</p>



<pre class="wp-block-code"><code># Every user's crontab, not just root's
for u in $(cut -d: -f1 /etc/passwd); do echo "== $u"; crontab -l -u "$u" 2&gt;/dev/null; done

# systemd timers, where a lot of "cron" actually lives now
systemctl list-timers --all

# WordPress: what is scheduled, and is anything overdue?
wp cron event list --fields=hook,next_run_relative,recurrence
wp config get DISABLE_WP_CRON</code></pre>



<p class="wp-block-paragraph">Run that on both servers and diff the output. That comparison is the whole check, and it takes two minutes.</p>



<p class="wp-block-paragraph">The specific traps worth knowing:</p>



<ul class="wp-block-list">
<li><strong>The old server&#8217;s cron is still running.</strong> Nobody stops it, because &#8220;keep it up for 48 hours&#8221; sounded like keep everything up. Now your nightly export runs twice, your reminder emails send twice, and your billing job runs against a database nobody is reading. Disable cron on the old box at cutover, before you disable anything else.</li>

<li><strong>WP-Cron was disabled in config and the replacement never came across.</strong> If <code>DISABLE_WP_CRON</code> is true and no system cron hits <code>wp-cron.php</code>, every scheduled task stops silently. Scheduled posts sit at &#8220;Missed schedule&#8221;, backups stop, WooCommerce action queues back up. The site itself works perfectly.</li>

<li><strong>The cron URL is wrong in a way that still returns success.</strong> A job calling an <code>http://</code> URL that now redirects to HTTPS gets a 3xx, exits zero, and looks healthy in the log while the task never executes. Same story for a cron still pointing at the old hostname or an old absolute path.</li>

<li><strong>Backups.</strong> The job may have moved and still be writing to storage credentials, a mount, or a destination path that only existed on the old host. Don&#8217;t check that the backup job ran. Check that a file arrived at the destination, and that it&#8217;s the size you expect.</li>

<li><strong>Log rotation.</strong> A fresh server often has no rotation configured for application logs. Nothing goes wrong for three days, then the disk fills at 2am and every write fails at once.</li>
</ul>



<h2 class="wp-block-heading">Hour 24 to 48: the delayed detonations</h2>



<p class="wp-block-paragraph">These are the checks with the longest detection latency, which makes them the ones most worth doing deliberately.</p>



<h3 class="wp-block-heading">Certificate renewal, not the certificate</h3>



<p class="wp-block-paragraph">A copied certificate is valid for weeks. What breaks in a migration is the machinery that replaces it: the webroot path in the renewal config no longer exists, the ACME account keys weren&#8217;t copied, port 80 is closed on the new firewall, or a WAF is now intercepting the challenge path. None of that surfaces until the renewal window opens, long after everyone has moved on.</p>



<pre class="wp-block-code"><code># What does certbot think it manages, and how does it validate?
certbot certificates

# The only check that matters: will the NEXT renewal succeed?
certbot renew --dry-run

# Is anything actually going to run it?
systemctl list-timers | grep -i certbot</code></pre>



<p class="wp-block-paragraph">The dry run validates against Let&#8217;s Encrypt&#8217;s staging environment, so it exercises the real challenge path without consuming your issuance quota. Resist the urge to fix a failure with a forced renewal: Let&#8217;s Encrypt applies rate limits per registered domain and a separate limit on identical certificates, and burning through them during a migration is how a fixable config problem turns into a week of waiting.</p>



<h3 class="wp-block-heading">Mail identity on an address with no history</h3>



<p class="wp-block-paragraph">If the new server sends mail, it&#8217;s sending from an IP that receiving systems have never seen. Authentication correctness and reputation are two separate problems and they fail differently. Authentication failures bounce or land in spam immediately. Reputation problems look like intermittent delivery to one provider and normal delivery to another.</p>



<pre class="wp-block-code"><code># Reverse DNS for the new sending IP
dig +short -x 203.0.113.10

# Does SPF still authorise everything that sends for you?
dig +short TXT example.com

# DKIM selector (replace 'selector') and DMARC policy
dig +short TXT selector._domainkey.example.com
dig +short TXT _dmarc.example.com</code></pre>



<p class="wp-block-paragraph">Send a test message to an address at a major provider and read the full headers. The Authentication-Results header tells you whether SPF, DKIM and DMARC each passed, and crucially whether the passing domain aligns with the visible From address. SPF can pass for the envelope sender and still fail DMARC when the From domain differs, which is a failure mode you cannot see from the sending side.</p>



<p class="wp-block-paragraph">Leave the old server&#8217;s SPF authorisation and DKIM public key published through the overlap. Mail that was queued or delayed on the old path still needs to verify. Sources differ on how long to keep them, and the honest answer is: until you have confirmed nothing is still sending or retrying from the old host.</p>



<p class="wp-block-paragraph">If transactional mail matters to the business, the migration is a reasonable moment to stop sending it from a web server at all. Handing it to a dedicated provider such as Postmark, Mailgun or Amazon SES moves reputation off an address you&#8217;ve just changed and gives you delivery logs you can actually query.</p>



<h3 class="wp-block-heading">Search crawl, redirects and staging leftovers</h3>



<p class="wp-block-paragraph">Crawlers hit harder than usual after a move, because they&#8217;re re-checking old URLs and discovering new ones at the same time. That surge is also your best early warning: it surfaces broken redirects faster than your own testing will.</p>



<ul class="wp-block-list">
<li>Fetch <code>robots.txt</code> from the live site and read it. A staging <code>Disallow: /</code> that survives cutover is the single most damaging thing on this list.</li>

<li>View source on several page types and check for a leftover noindex tag and for canonical tags pointing at a staging hostname.</li>

<li>Watch 404s in the access log rather than in a crawler report. It&#8217;s your missed-redirects list, sorted by how much anyone cares.</li>

<li>Leave the old URLs crawlable through the redirect. Blocking them means the redirects are never seen.</li>
</ul>



<h3 class="wp-block-heading">Resource behaviour under real traffic</h3>



<p class="wp-block-paragraph">Staging never told you what the process pool does at peak. Twenty-four hours of production traffic does.</p>



<pre class="wp-block-code"><code># Status code distribution since cutover
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

# Disk, and the one people forget
df -h
df -i</code></pre>



<p class="wp-block-paragraph">Compare the ratio of 5xx to total requests against the old server&#8217;s logs for the same weekday. Absolute numbers mislead here; the ratio doesn&#8217;t. Watch memory across a full daily cycle rather than at a single moment, because the interesting shape is a slow climb into a limit, not a spike.</p>



<h2 class="wp-block-heading">Before you decommission the old server</h2>



<p class="wp-block-paragraph">The old server is the only copy of some things, and none of them are in your backup.</p>



<ul class="wp-block-list">
<li>Access and error logs covering the cutover window, which is your evidence for anything that surfaces next month.</li>

<li>Mail spools, and any messages that arrived after the MX change.</li>

<li>The full crontab set, exported to a file rather than a screenshot.</li>

<li>Web server, PHP and database configuration, so you can diff a behavioural difference you notice later.</li>

<li>A final database dump taken after write traffic stopped.</li>
</ul>



<p class="wp-block-paragraph">Keep the TTL low until you&#8217;ve decided you&#8217;re staying. Raising it back to normal is the last step of the migration, not the first thing you do once the site loads. Another month on an InterServer or Contabo VPS is cheap next to reconstructing a week of orders, and the overlap is the only period in which rolling back is still a five-minute operation.</p>



<h2 class="wp-block-heading">Troubleshooting: three symptoms and what they usually mean</h2>



<p class="wp-block-paragraph"><strong>&#8220;It works for me but not for them.&#8221;</strong> Almost always a resolution split. Get the affected user to report what their resolver returns, then check whether the answer is stale DNS, a corporate resolver ignoring your TTL, or a CDN edge still holding the old origin. If you use Cloudflare or a similar proxy in front, remember the origin change and the DNS change are two separate things and only one of them propagates.</p>



<p class="wp-block-paragraph"><strong>&#8220;The site is fine but our mail goes to spam now.&#8221;</strong> Work in this order: reverse DNS for the sending IP, then SPF including the new address, then DKIM signing on the new host, then DMARC alignment with the visible From domain. Only after all four pass should you start thinking about reputation and volume, and reputation on a new IP is a matter of ramping gradually rather than something you can configure.</p>



<p class="wp-block-paragraph"><strong>&#8220;Random 500s that never happened in staging.&#8221;</strong> Look at limits before you look at code. Process pool size, memory limits, database connection caps, open file limits, and file ownership on directories the application writes to. Permission problems after an rsync or archive restore are common and produce errors that look nothing like permission problems from the front end.</p>



<h2 class="wp-block-heading">Common mistakes in the first 48 hours</h2>



<ul class="wp-block-list">
<li>Treating a green uptime check as verification. It confirms one URL returns 200. It says nothing about mail, cron, writes or checkout.</li>

<li>Leaving the old server fully operational, cron and all, instead of leaving it reachable but inert.</li>

<li>Testing while a hosts file entry is still in place.</li>

<li>Raising the TTL back to normal on day one, which throws away your fast rollback.</li>

<li>Checking that the certificate is valid instead of checking that renewal works.</li>

<li>Silencing monitoring for the maintenance window and forgetting to unsilence it.</li>

<li>Declaring the migration done before a full daily cron cycle has completed on the new server.</li>
</ul>



<h2 class="wp-block-heading">If you only do five things</h2>



<ol class="wp-block-list">
<li>Stop writes and cron on the old server, but keep it reachable.</li>

<li>Run <code>certbot renew --dry-run</code> and confirm a timer or cron will actually invoke it.</li>

<li>Send a real test email to a major provider and read the Authentication-Results header.</li>

<li>Diff the scheduled jobs on both servers, then confirm one real backup file landed at its destination.</li>

<li>Grep the old server&#8217;s access log for POST requests after the cutover, and reconcile anything you find.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">How long should I keep the old server running after a migration?</h3>



<p class="wp-block-paragraph">Long enough that no resolver anywhere is still handing out the old address, plus enough margin to reconcile data and pull logs. For most sites that&#8217;s a few days. The cost of the extra billing period is trivial compared with losing your rollback path, but keep it inert: reachable, not writable, with cron disabled.</p>



<h3 class="wp-block-heading">Why is my site fine but email broken after a server migration?</h3>



<p class="wp-block-paragraph">Because web and mail are validated by different systems using different records. The web path only needs an A record and a certificate. Mail needs the MX record, reverse DNS on the sending IP, SPF listing that IP, DKIM signing configured on the new host, and DMARC alignment with your From domain. A migration can satisfy the first set completely and break every item in the second.</p>



<h3 class="wp-block-heading">What should I check first if something looks wrong right after cutover?</h3>



<p class="wp-block-paragraph">Establish which server you are talking to before you diagnose anything else. Use <code>curl --resolve</code> to pin the hostname to the new IP and compare that response with what you get through normal DNS. Half of post-cutover panic is people debugging the old server.</p>



<h3 class="wp-block-heading">Can I lose form submissions or orders during DNS propagation?</h3>



<p class="wp-block-paragraph">Yes, and it&#8217;s the most common serious loss in a migration. Visitors on cached DNS reach the old application, which writes to the old database. Nothing errors. Proxying the old server to the new backend removes the problem entirely; short of that, freeze writes during the overlap and check the old access log for POST requests before decommissioning.</p>



<h3 class="wp-block-heading">Why did my SSL certificate stop renewing weeks after the migration?</h3>



<p class="wp-block-paragraph">The certificate itself migrated fine; the renewal configuration didn&#8217;t. Common causes are a webroot path that no longer exists, an ACME account directory that wasn&#8217;t copied, port 80 blocked by the new firewall, a proxy or WAF intercepting the challenge path, or no timer scheduled to run the renewal at all. A dry run within 48 hours catches all of these.</p>



<h3 class="wp-block-heading">Do I need to tell search engines about a server migration?</h3>



<p class="wp-block-paragraph">If the URLs are unchanged and only the hosting moved, no announcement is required. Your job is to make sure crawlers can reach the new server, that nothing carries a staging noindex or a blocking robots rule, and that canonical tags point at live URLs. If the domain or URL structure changed, that&#8217;s a different migration with its own redirect and site-move process.</p>



<h3 class="wp-block-heading">What monitoring should I add specifically for the 48-hour window?</h3>



<p class="wp-block-paragraph">Add checks for the things a homepage ping cannot see: a synthetic transaction through a real form or checkout, a heartbeat from each scheduled job, certificate expiry, and a content check that asserts the right text is present rather than just a 200 status. Most uptime platforms handle the first and last; heartbeat monitoring for cron is the one people skip and the one that catches the silent failures.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A migration doesn&#8217;t fail when the site goes down. It fails when the site stays up and something behind it stops, quietly, in a way nobody is looking for. Every item on this post-migration checklist exists because the failure it catches produces no visible symptom on the front end.</p>



<p class="wp-block-paragraph">So work the 48 hours in order of detection latency. Prove which server you&#8217;re on, close the write path on the old one, diff the scheduled jobs, dry-run the certificate renewal, and read the headers on a real test email. Then, and only then, raise the TTL and let the old box go.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on a cutover?</h2>



<p class="wp-block-paragraph">Migrations are one of the few pieces of infrastructure work where the review matters more than the execution. Things I help with:</p>



<ul class="wp-block-list">
<li>Planning the cutover so the write path is closed on the old server rather than left open, including proxy-forward setups for transactional sites.</li>

<li>Running the 48-hour verification pass on a migration someone else performed, and telling you plainly what&#8217;s still broken.</li>

<li>Repairing certificate renewal, ACME challenge paths and reload hooks that stopped working after the move.</li>

<li>Fixing mail on a new IP: reverse DNS, SPF, DKIM signing and DMARC alignment, or moving transactional mail off the web server entirely.</li>

<li>Rebuilding scheduled jobs, backup destinations and log rotation on the new host, with heartbeat monitoring so silence becomes an alert.</li>

<li>Reconciling data written to the old server during the split window, before anything gets deleted.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re mid-migration or just came out of one, send me something concrete: the old server&#8217;s access log around the cutover, your crontab, or the headers from a test email. That&#8217;s usually enough to tell you what&#8217;s wrong.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/post-migration-checklist-48-hours/">Post-Migration Checklist: What Breaks Quietly in the First 48 Hours</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://john-nessime.com/blog/devops/post-migration-checklist-48-hours/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>Oracle to Aurora Migration: The Failures Nobody Catches Until Cutover</title>
		<link>https://john-nessime.com/blog/devops/oracle-to-aurora-migration/</link>
					<comments>https://john-nessime.com/blog/devops/oracle-to-aurora-migration/#respond</comments>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS DMS]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cutover Planning]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Database Migration]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Oracle]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Schema Conversion]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=218</guid>

					<description><![CDATA[<p>Most Oracle to Aurora migrations don't fail loudly. They finish green, pass validation, and quietly hand you truncated LOBs, sequences stuck at 1, and empty strings that used to be NULL. Here are the four failure families to check before you cut over, with the SQL and task settings that catch them.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/oracle-to-aurora-migration/">Oracle to Aurora Migration: The Failures Nobody Catches Until Cutover</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Three weeks after cutover, someone in finance sends you a spreadsheet. Forty-one invoice records have a description field that stops mid-sentence. Not blank, not garbled. Just shorter than it used to be.</p>



<p class="wp-block-paragraph">So you open the AWS DMS console. The task is green. Full load completed, change data capture running, zero errors. Data validation passed on every table in scope.</p>



<p class="wp-block-paragraph">Both of those things are true at the same time, and that is the part that catches people out. An Oracle to Aurora migration almost never fails with a stack trace and a rollback. It fails with a green dashboard and a handful of rows that are subtly, permanently wrong.</p>



<p class="wp-block-paragraph">This post is not a walkthrough of the console wizard. AWS documents that well enough. It covers the four families of failure I would go looking for before signing off on an Oracle to Aurora migration: large objects that arrive truncated, change capture that records less than you assume, the semantic gaps between Oracle and PostgreSQL that compile cleanly and behave differently, and the database state that DMS simply does not move for you. Then how to validate properly, and what to do when a task stalls.</p>



<h2 class="wp-block-heading">Pick the Aurora engine before you pick the migration tool</h2>



<p class="wp-block-paragraph">Amazon Aurora comes in a PostgreSQL-compatible edition and a MySQL-compatible edition, and the choice shapes everything downstream. Do not let it get decided by whoever on the team happens to like MySQL.</p>



<p class="wp-block-paragraph">Aurora PostgreSQL is the realistic default coming from Oracle, and the reason is procedural code. PL/SQL and PL/pgSQL are close enough that automated conversion gets you a long way: packages, cursors, exceptions, custom types, and analytic functions all have recognisable counterparts. PostgreSQL also has the <code>orafce</code> extension, which reimplements a chunk of Oracle&#8217;s built-in functions and packages inside a compatibility schema, so a lot of converted code can keep calling things it already calls.</p>



<p class="wp-block-paragraph">The honest case for Aurora MySQL: if the Oracle database is essentially a table store with the business logic living in the application tier, and your team already runs MySQL in production, you will move faster on the engine you can operate at 3am. Migration difficulty is real, but so is the cost of running a database nobody on the rota understands. If your schema has no packages, no triggers worth keeping and no PL/SQL, that argument holds.</p>



<p class="wp-block-paragraph">Where it falls apart is the moment you find twelve thousand lines of PL/SQL that someone will have to rewrite by hand. At that point the engine choice has stopped being an operational preference and become a rewrite project.</p>



<h3 class="wp-block-heading">Schema conversion: SCT or the DMS console</h3>



<p class="wp-block-paragraph">There are two ways to convert the schema. The AWS Schema Conversion Tool is a desktop Java application that connects to both databases, reads the source objects, and emits target DDL plus an assessment report of everything it could not convert. DMS Schema Conversion is the same idea run from the DMS console using instance profiles and data providers, with no local install and no JDBC drivers to manage.</p>



<p class="wp-block-paragraph">Start with the console version. It is less setup and the assessment report is the thing you actually want early: a list of objects with action items, so you can size the manual work before committing to a date. Fall back to the desktop tool when you need something the console flow does not cover, such as converting application SQL files or SQL*Plus scripts.</p>



<p class="wp-block-paragraph">Either way, treat the assessment report as a project plan, not a progress bar. The red items are the schedule.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: LOB columns that arrive shorter than they left</h2>



<p class="wp-block-paragraph">This is the invoice description problem, and it is the single most common way an Oracle to Aurora migration loses data without telling anyone loudly.</p>



<p class="wp-block-paragraph">DMS gives you three ways to handle CLOB, NCLOB and BLOB columns:</p>



<ul class="wp-block-list">
<li><strong>Full LOB mode</strong> moves every LOB regardless of size. DMS has no idea how big they are, so it moves them one at a time, piece by piece. Correct, and slow.</li>



<li><strong>Limited LOB mode</strong> asks you for a maximum size up front. DMS pre-allocates memory and loads LOBs in bulk, which is dramatically faster. Anything larger than the limit is <em>truncated to the limit</em>, and a warning goes into the task log.</li>



<li><strong>Inline LOB mode</strong> sets a threshold. LOBs under it go inline with the row; larger ones fall back to the full LOB path with a lookup against the source. Good when most of your LOBs are small and a few are not.</li>
</ul>



<p class="wp-block-paragraph">Read the middle one again. Limited LOB mode is the default answer to &#8220;the full load is too slow&#8221;, it is the mode most tutorials use, and it silently clips your data. The warning lands in the task log, mixed in with everything else, and nobody is reading task logs line by line on a table with four million rows.</p>



<p class="wp-block-paragraph">Worse, the task still reports success. Truncation is not an error condition in DMS. It is expected behaviour for the mode you asked for.</p>



<h3 class="wp-block-heading">Measure before you set the limit</h3>



<p class="wp-block-paragraph">The fix is boring: go and find out how big your LOBs actually are, per column, on the source. Run this against Oracle for every LOB column in scope.</p>



<pre class="wp-block-code"><code>-- Largest LOB values in a single column, biggest first.
-- dbms_lob.getlength returns length in characters for CLOB/NCLOB,
-- bytes for BLOB. Check the top rows, not the average.
SELECT dbms_lob.getlength(description) AS lob_length
FROM   app.invoices
ORDER  BY dbms_lob.getlength(description) DESC
FETCH  FIRST 10 ROWS ONLY;

-- Same thing expressed in KB, which is the unit the DMS
-- "Max LOB size (K)" setting expects.
SELECT MAX(dbms_lob.getlength(description)) / 1024 AS max_kb
FROM   app.invoices;</code></pre>



<p class="wp-block-paragraph">Two things to know once you have the numbers.</p>



<p class="wp-block-paragraph">First, with Oracle as a source, DMS treats LOBs as VARCHAR data wherever it can, because bulk-fetching them is much faster than the LOB API. Oracle&#8217;s VARCHAR ceiling is 32K, so a limited LOB size under 32K is where the mode performs best. If your real maximum is comfortably under that, limited LOB mode is genuinely the right call.</p>



<p class="wp-block-paragraph">Second, memory. During full load, DMS pre-allocates roughly the max LOB size multiplied by the commit rate multiplied by the number of LOB columns. Push the limit up and that product grows fast. When the replication instance cannot allocate it, it starts swapping, and your &#8220;faster&#8221; mode becomes slower than full LOB mode. If limited LOB mode is dragging, drop the commit rate before you drop the size limit.</p>



<h3 class="wp-block-heading">Split the LOB tables into their own task</h3>



<p class="wp-block-paragraph">The pattern I reach for first: one task with limited LOB mode for the bulk of the schema, a second task for the handful of tables with genuinely large LOBs, using full or inline mode. You can also override task-level LOB settings per table inside the table mapping rules, which keeps it to a single task if you prefer.</p>



<pre class="wp-block-code"><code>{
  "TargetMetadata": {
    "SupportLobs": true,
    "FullLobMode": false,
    "LimitedSizeLobMode": true,
    "LobMaxSize": 16,
    "InlineLobMaxSize": 0,
    "LobChunkSize": 64,
    "BatchApplyEnabled": false
  }
}</code></pre>



<p class="wp-block-paragraph">That is limited mode with a 16 KB ceiling. To use inline mode instead, set <code>FullLobMode</code> to true and give <code>InlineLobMaxSize</code> a non-zero value, which is the threshold below which LOBs travel inline.</p>



<p class="wp-block-paragraph">One requirement that bites during ongoing replication: for CDC, tables with LOB columns need a primary key, because DMS looks the LOB value up in the source rather than reading it out of the redo stream. Full load does not care. CDC does. A LOB table with no primary key will load fine and then fail to replicate changes.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family two: CDC that captures less than you think</h2>



<p class="wp-block-paragraph">Ongoing replication is what buys you a short cutover window. It is also where the assumptions hide.</p>



<p class="wp-block-paragraph">DMS reads Oracle&#8217;s redo logs. For that to work at all, the source has to be in ARCHIVELOG mode with enough retention that DMS can still find the logs it needs after a pause, a network blip or an overnight stall. Retention is the one people get wrong: a task that stops for six hours against a two-hour retention window does not resume, it fails, and you restart from a full load.</p>



<h3 class="wp-block-heading">Supplemental logging is not optional</h3>



<p class="wp-block-paragraph">Oracle&#8217;s redo logs, by default, contain enough information for Oracle to recover the database. That is not the same as enough information to reconstruct a row-level change for a different database. Supplemental logging is what closes that gap.</p>



<pre class="wp-block-code"><code>-- Minimal supplemental logging at the database level.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

-- Identification key logging: writes every column of a row's
-- primary key into redo on UPDATE, even when the key didn't change.
-- Without this, DMS cannot reliably match the row on the target.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;

-- Per-table alternative, if you only need a subset of tables
-- and want to keep the source overhead down.
ALTER TABLE app.invoices ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;</code></pre>



<p class="wp-block-paragraph">The case that actually bites is a table with no primary key and no unique index. For those, key-level logging gives DMS nothing to work with, and it needs supplemental logging on <em>all</em> columns so the before-and-after image of the row lands in redo. Miss that, and DMS cannot build a safe UPDATE or DELETE statement for the target. You will not notice during full load. You will notice when an update on the source produces no change on the target, or changes the wrong row.</p>



<p class="wp-block-paragraph">DMS ships premigration assessments that check exactly this. Run them. There are individual checks for supplemental logging on keyed tables, supplemental logging on all columns for unkeyed tables, ARCHIVELOG configuration, the CDC method in use, target permissions, and triggers enabled on the target. It takes minutes and it replaces a category of guesswork.</p>



<h3 class="wp-block-heading">LogMiner or Binary Reader</h3>



<p class="wp-block-paragraph">DMS can read redo two ways. Oracle LogMiner is an Oracle-side API; DMS calls it and Oracle does the parsing. Binary Reader is DMS&#8217;s own parser reading the redo files directly.</p>



<p class="wp-block-paragraph">If you specify nothing, you get LogMiner. That is usually correct. It needs fewer privileges, it handles ASM without extra plumbing, and it supports Oracle features like encryption and compression that the direct parser does not cover. The grants are small:</p>



<pre class="wp-block-code"><code>GRANT EXECUTE ON DBMS_LOGMNR TO dms_user;
GRANT SELECT ON V_$LOGMNR_LOGS TO dms_user;
GRANT SELECT ON V_$LOGMNR_CONTENTS TO dms_user;
GRANT LOGMINING TO dms_user;</code></pre>



<p class="wp-block-paragraph">Binary Reader earns its keep in one situation: a source generating redo faster than LogMiner can chew through it, or several concurrent migration tasks against the same database. LogMiner runs inside the source instance, so its CPU cost lands on the database you are trying not to disturb. Binary Reader shifts that work to the replication instance.</p>



<p class="wp-block-paragraph">The trade-off is setup. Binary Reader needs additional privileges and file-level access to the redo logs, which means either a directory the replication instance can reach or a copy step. If your source is a busy production Oracle instance and CDC latency is climbing steadily rather than spiking, that is the signal to switch. Otherwise stay on LogMiner and spend the complexity budget elsewhere.</p>



<p class="wp-block-paragraph">Whichever you pick, put CDC latency on a dashboard for the whole replication window, not just cutover day. The DMS task metrics publish to CloudWatch, and pulling them into whatever you already run for alerting, whether that is Grafana Cloud, Datadog or New Relic, is worth the twenty minutes. Source latency climbing while target latency stays flat means you are not reading redo fast enough. Both climbing together usually means the target cannot apply fast enough.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family three: the semantics Oracle and PostgreSQL do not share</h2>



<p class="wp-block-paragraph">Schema conversion produces code that compiles. Compiling is not the same as behaving identically, and this is where a migration goes wrong in ways no tool flags.</p>



<h3 class="wp-block-heading">Empty string is not NULL</h3>



<p class="wp-block-paragraph">Oracle treats the empty string and NULL as the same thing. PostgreSQL does not. This one difference propagates into comparisons, concatenation, unique constraints, and any procedural code that checks whether a value is &#8220;empty&#8221;.</p>



<pre class="wp-block-code"><code>-- PostgreSQL
SELECT '' IS NULL;        -- false
SELECT NULL || 'abc';     -- NULL
SELECT '' || 'abc';       -- abc

-- Oracle
SELECT CASE WHEN '' IS NULL THEN 'yes' ELSE 'no' END FROM dual;  -- yes
SELECT NULL || 'abc' FROM dual;                                  -- abc
SELECT '' || 'abc' FROM dual;                                    -- abc</code></pre>



<p class="wp-block-paragraph">Every one of those lines is a place a report can quietly change its numbers. Unique constraints are worse. Oracle stores an empty string as NULL, and NULLs are not compared for uniqueness, so it will happily accept many rows with an empty value in a uniquely constrained column. PostgreSQL sees zero-length strings, which are equal to each other, so the second one is rejected. Same schema, same data, different outcome, and the failure surfaces as an application error nobody can reproduce locally.</p>



<p class="wp-block-paragraph">There is no switch for this. Either the application changes, or you wrap the affected functions, or you normalise on write with a trigger or a check constraint. Decide which, per column, before cutover. The orafce extension helps with function behaviour but it is not a global compatibility mode, and enabling the parts that change NULL semantics has its own consequences.</p>



<h3 class="wp-block-heading">The rest of the list</h3>



<ul class="wp-block-list">
<li><strong>ROWID</strong> has no PostgreSQL equivalent. DMS Schema Conversion can emulate it with a bigint or a character varying column, but if application code depends on ROWID ordering or reuse semantics, emulation will not save you.</li>



<li><strong>SYSDATE and time zones.</strong> Conversion settings let you choose whether to emulate Oracle&#8217;s time zone handling or use native PostgreSQL behaviour. Native is faster. It is only safe if the database and the application genuinely run in the same time zone.</li>



<li><strong>TO_CHAR, TO_DATE, TO_NUMBER format masks.</strong> Oracle accepts parameters PostgreSQL does not. The converter emulates them by default, which is correct and slower. Turning emulation off is a per-codebase decision, not a global one.</li>



<li><strong>DUAL</strong> does not exist. PostgreSQL does not need a FROM clause, so most references just get dropped, but any dynamic SQL that builds a query string around DUAL needs finding.</li>



<li><strong>Synonyms</strong> have no direct equivalent. Search path adjustments and views cover most cases.</li>



<li><strong>Materialized views</strong> can be converted to real materialized views or to plain tables. The refresh semantics differ, so check what the reporting layer expects.</li>



<li><strong>NUMBER without a scale.</strong> An Oracle NUMBER column with no scale specified converts to something with a default, and if the source holds values wider than that default you get rounding or overflow. There is a premigration assessment specifically for this.</li>
</ul>



<p class="wp-block-paragraph">One data-level quirk worth knowing: if an Oracle column contains a NULL character (hex U+0000), DMS converts it to a space (U+0020) on a PostgreSQL target, because PostgreSQL cannot store it in a text column. Rare, but if you are migrating anything that stuffed binary into a VARCHAR, that is a real change to your data.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family four: the state DMS never moves</h2>



<p class="wp-block-paragraph">DMS moves rows. It does not move everything that makes a database work, and the gaps are not obvious until you point traffic at the new cluster.</p>



<h3 class="wp-block-heading">Sequences</h3>



<p class="wp-block-paragraph">Schema conversion creates the sequences. DMS copies the rows. Nothing advances the sequence counters to match the data that just arrived, so your target sequences sit at their starting value while the table already contains eight million rows.</p>



<p class="wp-block-paragraph">The first insert after cutover collides with an existing primary key. Then the second. Then every write for the next eight million attempts.</p>



<p class="wp-block-paragraph">Fix this after you stop replication and before you open traffic, not earlier, or you will do it twice.</p>



<pre class="wp-block-code"><code>-- Advance a sequence to match the data already loaded.
-- Run once per sequence, after CDC has stopped.
SELECT setval('app.invoices_id_seq',
              (SELECT max(id) FROM app.invoices));

-- Confirm before you open the doors.
SELECT last_value FROM app.invoices_id_seq;</code></pre>



<h3 class="wp-block-heading">Triggers on the target</h3>



<p class="wp-block-paragraph">If your converted schema has triggers and they are enabled during the load, they fire on every row DMS inserts. Audit tables get populated with a migration&#8217;s worth of fake activity. Derived columns get recalculated against half-loaded data. Anything that writes to another table doubles your write volume.</p>



<p class="wp-block-paragraph">Disable target triggers for the load, re-enable at cutover. There is a premigration assessment that checks for enabled triggers on target tables in task scope, which is a decent backstop against forgetting.</p>



<h3 class="wp-block-heading">Secondary indexes and foreign keys</h3>



<p class="wp-block-paragraph">Every secondary index present during full load is an index being maintained row by row while you are trying to move data as fast as possible. Foreign keys are worse, because DMS does not guarantee parent-before-child load order across tables.</p>



<p class="wp-block-paragraph">Standard sequence: drop or disable secondary indexes and foreign key constraints before full load, recreate them afterwards, then let CDC catch up. Keep primary keys, since CDC needs them. Script the recreation and check it into version control alongside everything else, because rebuilding forty indexes from memory at 2am is not a plan.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Validation you can actually trust</h2>



<p class="wp-block-paragraph">DMS data validation compares source and target row by row and reports mismatches. It supports Oracle and PostgreSQL-compatible endpoints in both directions, so an Oracle to Aurora PostgreSQL path is covered. Turn it on.</p>



<p class="wp-block-paragraph">Then understand what it will not tell you.</p>



<ul class="wp-block-list">
<li>It needs a primary key or unique index on both sides. Unkeyed tables are not validated, and those are exactly the tables most likely to have gone wrong.</li>



<li>Collation and sort order differ between Oracle and PostgreSQL. When they disagree, validation reports failures on records that are actually fine, and the noise trains people to ignore the report.</li>



<li>It does not run when the migration uses custom filtering, or when you are consolidating several source databases into one target.</li>



<li>It costs real resources. Validation issues its own queries against both databases, on top of the migration traffic. Budget for that on the source.</li>



<li>For LOB columns it compares checksums rather than values, using DBMS_CRYPTO on the Oracle side.</li>
</ul>



<p class="wp-block-paragraph">Because validation runs alongside migration, a task that stops takes validation with it. Validation-only tasks solve that: same endpoints, same table mappings, no data movement, running independently of the migration task. Use one when the validation load is hurting the source and you want to run it off-peak, or when you want validation to survive a migration task restart.</p>



<p class="wp-block-paragraph">None of that replaces your own reconciliation. Before cutover, run business-level checks on both sides: rows per table, sums of every monetary column, min and max of every date column, distinct counts on anything you join on. Those catch things row comparison does not, because they answer the question the business actually asks.</p>



<p class="wp-block-paragraph">And specifically for the LOB problem, compare maximum lengths, not just row counts.</p>



<pre class="wp-block-code"><code>-- On Aurora PostgreSQL, after the load.
-- Compare this against dbms_lob.getlength on the Oracle source.
SELECT count(*)                 AS row_count,
       max(length(description)) AS max_len,
       count(*) FILTER (WHERE description IS NULL) AS null_count
FROM   app.invoices;</code></pre>



<p class="wp-block-paragraph">If the target maximum is suspiciously round, exactly 16384 or exactly 32768, you have found your truncation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting an Oracle to Aurora migration that stalls</h2>



<h3 class="wp-block-heading">Full load runs, then one table sits at zero</h3>



<p class="wp-block-paragraph">Almost always LOBs. Full LOB mode moves large objects one at a time with a source lookup per value, and on a table with a million large CLOBs that is a very long time with no visible progress. Check the table statistics for that table, then check whether it has LOB columns. Move it to its own task with limited or inline mode and an appropriate size.</p>



<h3 class="wp-block-heading">CDC latency climbs and never comes down</h3>



<p class="wp-block-paragraph">Look at which latency is climbing. Source latency means redo is arriving faster than DMS can read it, which is the LogMiner to Binary Reader conversation, or a redo volume problem on the source. Target latency means Aurora cannot apply fast enough, which points at indexes still present, triggers still enabled, or an undersized writer instance.</p>



<h3 class="wp-block-heading">Task fails on restart with missing archived logs</h3>



<p class="wp-block-paragraph">Retention. The logs DMS needed were removed before it came back. Increase archived log retention on the source to comfortably exceed your worst realistic outage, then restart from a fresh full load for the affected tables. There is no recovering the gap.</p>



<h3 class="wp-block-heading">Updates on the source do not appear on the target</h3>



<p class="wp-block-paragraph">Supplemental logging, nine times out of ten. Check whether the affected table has a primary key or unique index, and whether logging is configured at the right level for that answer. The premigration assessments cover both cases.</p>



<h3 class="wp-block-heading">Errors about a unique index on the target</h3>



<p class="wp-block-paragraph">DMS does not support replication to a table with a unique index built on a <code>coalesce</code> expression. If schema conversion produced one, or someone added it by hand to work around the empty-string problem, replication to that table will not work. Restructure the constraint.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Accepting the default limited LOB size without measuring the source. This is the truncation bug, and it has a specific, preventable cause.</li>



<li>Treating a green task and a passed validation as proof of correctness. Both are true when data has been truncated.</li>



<li>Skipping the premigration assessments because the wizard let you continue without them.</li>



<li>Forgetting to advance sequences, then discovering it under production write load.</li>



<li>Leaving triggers enabled on the target during full load and polluting audit tables.</li>



<li>Assuming schema conversion handled empty strings. It converted the syntax, not the semantics.</li>



<li>Setting archived log retention to the length of a normal outage rather than a bad one.</li>



<li>Doing the first end-to-end rehearsal on cutover night.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ol class="wp-block-list">
<li><strong>Run the assessment report first and size the manual work from it.</strong> The red items are your timeline, and they are better known in week one than week nine.</li>



<li><strong>Inventory LOB columns before configuring anything.</strong> Maximum size per column, in KB, written down.</li>



<li><strong>Separate the LOB-heavy tables into their own task.</strong> Different tables want different modes, and one global setting will always be wrong for someone.</li>



<li><strong>Enable supplemental logging deliberately, per table class.</strong> Key-level for keyed tables, all-columns for unkeyed ones.</li>



<li><strong>Drop secondary indexes and foreign keys for the load.</strong> Script the recreation, version it, test it.</li>



<li><strong>Rehearse the whole cutover at least twice</strong>, including the sequence advance and the trigger re-enable. A scratch environment for the dry run does not have to be expensive: a temporary Aurora cluster you tear down afterwards, or a cheap dedicated box from somewhere like InterServer or Hetzner running a trimmed source copy, both work.</li>



<li><strong>Run your own reconciliation alongside DMS validation.</strong> Counts, sums, min and max dates, maximum LOB lengths.</li>



<li><strong>Keep the Oracle source running and readable for a while after cutover.</strong> The bugs that surface at week three are the ones you can only diagnose by comparing against the original.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Should I migrate Oracle to Aurora PostgreSQL or Aurora MySQL?</h3>



<p class="wp-block-paragraph">Aurora PostgreSQL for anything with meaningful PL/SQL, because PL/pgSQL is close enough that automated conversion does most of the work, and the orafce extension covers a chunk of the rest. Aurora MySQL is defensible when the schema is plain tables, the logic lives in the application, and your team already operates MySQL. The deciding question is how many lines of procedural code you would have to rewrite by hand.</p>



<h3 class="wp-block-heading">Can AWS DMS migrate stored procedures and packages?</h3>



<p class="wp-block-paragraph">No. DMS moves data. Schema and code conversion is a separate job, handled by DMS Schema Conversion in the console or the AWS Schema Conversion Tool on the desktop. Both convert most objects automatically and produce an action-item list for the rest.</p>



<h3 class="wp-block-heading">Why is my DMS task green when data is missing?</h3>



<p class="wp-block-paragraph">Because truncation in limited LOB mode is expected behaviour, not an error. DMS writes a warning to the task log and carries on. Task status reflects whether the task is running, not whether the data is faithful. Compare maximum LOB lengths between source and target to catch it.</p>



<h3 class="wp-block-heading">Do I need downtime for an Oracle to Aurora migration?</h3>



<p class="wp-block-paragraph">Some, but it can be short. Full load plus CDC means the target stays current while you test, so the outage is limited to stopping writes, letting CDC drain, advancing sequences, re-enabling triggers, recreating indexes and repointing the application. Rehearse it and that usually fits a maintenance window rather than a weekend.</p>



<h3 class="wp-block-heading">Does DMS migrate sequences?</h3>



<p class="wp-block-paragraph">It copies row data, not sequence state. Schema conversion creates the sequence objects; you advance them yourself after stopping replication, using <code>setval</code> against the current maximum key value. Skipping this produces primary key collisions on the first write after cutover.</p>



<h3 class="wp-block-heading">LogMiner or Binary Reader for Oracle CDC?</h3>



<p class="wp-block-paragraph">LogMiner unless you have a reason. It is the default, needs fewer privileges, and handles ASM, encryption and compression without extra work. Move to Binary Reader when redo volume is high enough that LogMiner cannot keep up, when you are running several tasks against the same source, or when the CPU cost of parsing on the source database is unacceptable.</p>



<h3 class="wp-block-heading">Will DMS data validation catch every problem?</h3>



<p class="wp-block-paragraph">No. It needs a primary key or unique index on both sides, it can report false failures where Oracle and PostgreSQL collations disagree, and it does not run with custom filtering or multi-source consolidation. Treat it as one layer and add business-level reconciliation on top.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An Oracle to Aurora migration does not usually announce its failures. The task goes green, validation passes, and the damage shows up weeks later as a truncated field, a duplicate key, a report whose totals moved slightly.</p>



<p class="wp-block-paragraph">So build the migration around proving correctness rather than proving completion. Measure your LOBs before you set a limit. Configure supplemental logging for the tables that have no key, not just the ones that do. Assume the semantic gaps between Oracle and PostgreSQL are still there after conversion, because they are. Advance the sequences. Then reconcile at the level the business cares about, not just row by row.</p>



<p class="wp-block-paragraph">Green means the pipe is open. It does not mean what came out the other end is what went in.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Oracle to Aurora migration?</h2>



<p class="wp-block-paragraph">Most of the work on these projects is not running the wizard. It is finding the handful of things that will be wrong afterwards. That is the part I help with:</p>



<ul class="wp-block-list">
<li>Reviewing DMS task settings and table mappings for LOB truncation risk, per column, before you run a full load you cannot easily repeat.</li>



<li>Working through the schema conversion assessment report and turning the red items into an actual estimate rather than a wall of warnings.</li>



<li>Setting up supplemental logging and CDC correctly for tables with no primary key, including the LogMiner versus Binary Reader call for your redo volume.</li>



<li>Writing the cutover runbook: index recreation scripts, sequence advance, trigger re-enable, ordered, tested, and timed against a real rehearsal.</li>



<li>Building the reconciliation layer that sits on top of DMS validation, so you can sign off on the data with something better than a green tick.</li>



<li>Post-cutover triage when something on Aurora behaves differently from Oracle and nobody can work out which layer changed.</li>
</ul>



<p class="wp-block-paragraph">If you have a task settings JSON, an assessment report, or a validation failure log you cannot make sense of, send it over and I will tell you what I see in it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/oracle-to-aurora-migration/">Oracle to Aurora Migration: The Failures Nobody Catches Until Cutover</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://john-nessime.com/blog/devops/oracle-to-aurora-migration/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
	</channel>
</rss>
