<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Chain of Custody | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/chain-of-custody/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/chain-of-custody/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 31 Oct 2026 06:08:13 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Chain of Custody | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/chain-of-custody/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Compromised VPS Recovery: What You Can Actually Fix Without Reinstalling</title>
		<link>https://john-nessime.com/blog/system-administration/compromised-vps-recovery-without-reinstall/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 31 Oct 2026 06:08:13 +0000</pubDate>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Chain of Custody]]></category>
		<category><![CDATA[Cron]]></category>
		<category><![CDATA[Digital Forensics]]></category>
		<category><![CDATA[File Integrity Monitoring]]></category>
		<category><![CDATA[Incident Response]]></category>
		<category><![CDATA[Persistence Mechanisms]]></category>
		<category><![CDATA[Rescue Mode]]></category>
		<category><![CDATA[Rootkits]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[systemd Timers]]></category>
		<category><![CDATA[VPS]]></category>
		<category><![CDATA[Web Shell]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=496</guid>

					<description><![CDATA[<p>Killing the process removes the symptom. It does nothing about the SSH key on a service account or the timer pulling a fresh payload every twenty minutes. A working guide to compromised VPS recovery: how to judge the depth of an intrusion, keep the evidence, enumerate the access paths and persistence people miss, and decide honestly whether you can clean in place or need to rebuild.</p>
<p>The post <a href="https://john-nessime.com/blog/system-administration/compromised-vps-recovery-without-reinstall/">Compromised VPS Recovery: What You Can Actually Fix Without Reinstalling</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email from the provider is short and unfriendly. Outbound scanning detected on port 22, network access to your instance has been rate limited, respond within 24 hours. You SSH in, find a process you don&#8217;t recognise chewing a core, kill it, delete the binary out of <code>/tmp</code>, and write back saying it&#8217;s handled.</p>



<p class="wp-block-paragraph">Three days later you get the same email.</p>



<p class="wp-block-paragraph">That gap is the whole problem with compromised VPS recovery. Killing the process removes the symptom. It does nothing about the SSH key that got added to a service account, the systemd timer that pulls a fresh payload every twenty minutes, or the reason they got in the first time. You cleaned up artifacts. You never closed the access.</p>



<p class="wp-block-paragraph">This post is about what you can honestly fix in place, what you can&#8217;t, and how to tell the difference before you sink a weekend into a box you&#8217;re going to rebuild anyway: evidence handling, the access paths and persistence people miss, why package verification proves less than it looks, and the one in-place technique that genuinely holds up.</p>



<h2 class="wp-block-heading">What compromised VPS recovery actually means</h2>



<p class="wp-block-paragraph">There are two goals hiding inside &#8220;recover the server&#8221; and they pull in different directions.</p>



<ul class="wp-block-list"><li><strong>Restore service.</strong> Get the site back, stop the abuse traffic, keep the client happy. Fast, and you can do it on the same instance.</li><li><strong>Restore trust.</strong> Be able to say, with something behind it, that nothing on this machine is executing code you didn&#8217;t put there. Slow, and on a machine where someone had root, often impossible.</li></ul>



<p class="wp-block-paragraph">A reinstall is not a technical superstition. It&#8217;s the cheapest way to buy the second goal. You throw away every byte you can&#8217;t account for and start from a base image plus config you control. Cleaning in place buys you the first goal quickly and the second one slowly, sometimes never.</p>



<p class="wp-block-paragraph">So the default answer is still: rebuild from a known-good image, restore validated data, re-point DNS. What follows is for when that isn&#8217;t available today, or when you have to investigate before you&#8217;re allowed to destroy anything.</p>



<h2 class="wp-block-heading">Decide this first: how deep did they get</h2>



<p class="wp-block-paragraph">Everything else follows from this. The depth of the compromise decides whether in-place recovery is defensible or wishful thinking.</p>



<h3 class="wp-block-heading">Unprivileged application code</h3>



<p class="wp-block-paragraph">A web shell dropped in a document root, running as the PHP-FPM pool user or <code>www-data</code>. It can read what that user reads, write where that user writes, and make outbound connections. It cannot touch <code>/usr/bin</code> or install a kernel module.</p>



<p class="wp-block-paragraph">In-place recovery is reasonable here, as long as you can prove the blast radius stayed inside that user. That proof is the hard part, not the cleanup.</p>



<h3 class="wp-block-heading">A service account with a path upward</h3>



<p class="wp-block-paragraph">Same as above, but the account has a sudoers entry, write access to a systemd unit directory, membership in the <code>docker</code> group, or a writable script that root runs on a schedule. Treat this as a root compromise until you&#8217;ve disproved it, because all four of those are one command away from root.</p>



<h3 class="wp-block-heading">Root</h3>



<p class="wp-block-paragraph">Now the machine can lie to you. Root can replace <code>ps</code>, edit the package database, truncate logs, and set the immutable attribute on files so your own cleanup silently fails. Anything the live system tells you is a claim, not a fact.</p>



<p class="wp-block-paragraph">You can still investigate. You cannot honestly certify the result. If someone needs a clean bill of health, rebuild.</p>



<h3 class="wp-block-heading">Kernel</h3>



<p class="wp-block-paragraph">A loadable kernel module or an eBPF program that hides processes, files and network connections from userspace. Your tools return whatever the rootkit decides they should return. There is no in-place recovery here, only offline analysis of the disk and a rebuild.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Before you touch anything, keep the evidence</h2>



<p class="wp-block-paragraph">The first fifteen minutes decide how much you&#8217;ll be able to learn later. Most of what tells you how bad it is only exists while the box is running, and most of it gets destroyed by the instinctive reaction.</p>



<ol class="wp-block-list"><li><strong>Isolate at the edge, not from inside.</strong> Use your provider&#8217;s firewall or security group to cut everything except your own source address. Running <code>ip link set eth0 down</code> from an SSH session locks you out and does not stop a local timer from running.</li><li><strong>Take a snapshot.</strong> Every serious VPS provider offers one, and it costs almost nothing to keep for a couple of weeks. This is your only chance at a copy of the disk in its compromised state. Providers like Hetzner, Contabo, OVHcloud and InterServer all expose snapshots or images from the control panel.</li><li><strong>Collect volatile state before you reboot.</strong> Sockets, processes, open files and the mapping from process to on-disk binary all disappear the moment the box restarts.</li><li><strong>Copy the collection off the box.</strong> To your workstation or a bucket, not to another directory on the same instance.</li></ol>



<p class="wp-block-paragraph">A collection pass that fits in a couple of minutes:</p>



<pre class="wp-block-code"><code># Sockets with owning process, numeric, TCP and UDP
ss -tunap &gt; /tmp/ir-sockets.txt

# Process tree, full command lines
ps auxf &gt; /tmp/ir-ps.txt

# Where each PID's executable actually lives on disk.
# A deleted binary still running shows up as "(deleted)".
for p in /proc/[0-9]*; do
  echo "$p -&gt; $(readlink "$p/exe" 2&gt;/dev/null)"
done &gt; /tmp/ir-exe.txt

# Logins, including failed ones
last -F -f /var/log/wtmp &gt; /tmp/ir-last.txt
lastb -F                 &gt; /tmp/ir-lastb.txt 2&gt;/dev/null

# Journal for the suspect window
journalctl --since "7 days ago" --no-pager &gt; /tmp/ir-journal.txt</code></pre>



<p class="wp-block-paragraph">The <code>readlink</code> loop is the one people skip and then wish they hadn&#8217;t. A process whose executable was unlinked after launch is a strong signal on its own, and it&#8217;s the only place the original path survives.</p>



<p class="wp-block-paragraph">If <code>auditd</code> was already running, pull its logs too. If it wasn&#8217;t, note that for the rebuild rather than installing it now on a machine you don&#8217;t trust.</p>



<h2 class="wp-block-heading">Failure family 1: you removed the payload and left the door</h2>



<p class="wp-block-paragraph">This is the one that produces the second abuse email. Access mechanisms are boring, they don&#8217;t show up in <code>top</code>, and they survive every cleanup that focuses on running processes.</p>



<pre class="wp-block-code"><code># Authorized keys for every account that has a home directory
for h in $(getent passwd | cut -d: -f6 | sort -u); do
  [ -f "$h/.ssh/authorized_keys" ] &amp;&amp; echo "== $h" &amp;&amp; cat "$h/.ssh/authorized_keys"
done

# sshd may have been told to read keys from somewhere else entirely
grep -Ei 'AuthorizedKeysFile|AuthorizedKeysCommand|PermitRootLogin|PasswordAuthentication|^Match' 
  /etc/ssh/sshd_config /etc/ssh/sshd_config.d/*.conf 2&gt;/dev/null

# Any account with UID 0 other than root
awk -F: '($3 == 0) {print}' /etc/passwd

# Accounts that can actually log in
awk -F: '$7 !~ /(nologin|false)$/ {print $1, $7}' /etc/passwd

# Sudo grants, including drop-in files
cat /etc/sudoers; ls -la /etc/sudoers.d/</code></pre>



<p class="wp-block-paragraph">Two of these deserve more attention than they usually get. <code>AuthorizedKeysFile</code> pointing at a path outside a user&#8217;s home means your audit of <code>~/.ssh/authorized_keys</code> proves nothing. And a <code>Match</code> block near the bottom of <code>sshd_config</code> can quietly re-enable password authentication or root login for one user or one source address while the top of the file still looks locked down.</p>



<p class="wp-block-paragraph">Also check <code>/etc/pam.d/</code> for modules you don&#8217;t recognise. PAM is a legitimate extension point, which is exactly why it&#8217;s a good hiding place, and a single added line can accept any password for a given account.</p>



<h2 class="wp-block-heading">Failure family 2: persistence in the places you didn&#8217;t check</h2>



<p class="wp-block-paragraph">Cron gets checked. The rest usually doesn&#8217;t. MITRE ATT&amp;CK catalogues these as separate techniques for a reason: they&#8217;re independent, and finding one tells you nothing about the others.</p>



<pre class="wp-block-code"><code># Per-user crontabs and the system drop-in directories
for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u "$u" 2&gt;/dev/null; done
ls -la /etc/cron.d/ /etc/cron.hourly/ /etc/cron.daily/ /var/spool/cron/

# systemd: enabled units and every timer, including inactive ones
systemctl list-unit-files --type=service --state=enabled
systemctl list-timers --all
systemctl cat suspicious-unit.service   # shows the actual ExecStart

# Per-user systemd units survive logout when lingering is on
ls -la /root/.config/systemd/user/ /home/*/.config/systemd/user/ 2&gt;/dev/null
loginctl list-users

# One-shot at jobs, frequently overlooked
atq

# Dynamic linker hijacking
cat /etc/ld.so.preload 2&gt;/dev/null

# Shell and login hooks
ls -la /etc/profile.d/ /etc/update-motd.d/ 2&gt;/dev/null
grep -rn 'curl|wget|base64|/dev/tcp' /root/.bashrc /home/*/.bashrc 
  /etc/bash.bashrc 2&gt;/dev/null

# Unexpected setuid binaries and file capabilities
find / -xdev -perm -4000 -type f 2&gt;/dev/null
getcap -r / 2&gt;/dev/null</code></pre>



<p class="wp-block-paragraph"><code>/etc/ld.so.preload</code> is worth understanding rather than just grepping. Any library listed there is loaded into every dynamically linked process before anything else, which lets it intercept ordinary libc calls. That&#8217;s how a userspace rootkit makes <code>readdir</code> skip its own files and <code>ps</code> skip its own processes without touching the kernel. If that file exists on a machine where you didn&#8217;t put it, everything you&#8217;ve observed so far is suspect.</p>



<p class="wp-block-paragraph">The other one to internalise is timers. A cleanup that looks successful, followed by reinfection at a suspiciously round interval, is almost always a systemd timer or a cron entry you didn&#8217;t enumerate. <code>systemctl list-timers --all</code> matters because a timer that isn&#8217;t currently active is invisible without the flag.</p>



<h2 class="wp-block-heading">Failure family 3: verification that verifies less than you think</h2>



<p class="wp-block-paragraph">Package managers keep hashes of the files they installed, so you can ask the distro whether anything on disk drifted from what shipped.</p>



<pre class="wp-block-code"><code># RHEL, AlmaLinux, Rocky, Fedora: verify every installed package
rpm -Va

# Debian and Ubuntu: dpkg's own verify, no extra package needed
dpkg -V

# Optional on Debian/Ubuntu, checks more files than dpkg -V
debsums -s</code></pre>



<p class="wp-block-paragraph">In <code>rpm -Va</code> output, the character positions are the checks that failed: <code>5</code> is a changed digest, <code>S</code> size, <code>T</code> timestamp, <code>M</code> mode. A <code>c</code> after the flags marks a config file, and config files change legitimately all the time. A modified digest on something in <code>/usr/bin</code> or <code>/usr/sbin</code> does not.</p>



<p class="wp-block-paragraph">Now the caveats, because this is where people talk themselves into a false clean bill of health:</p>



<ul class="wp-block-list"><li>It only covers files that came from a package. A web shell in your document root, a binary in <code>/tmp</code>, a rogue systemd unit and anything you compiled yourself are all invisible to it.</li><li>The database and the verifying binary both live on the machine being verified. Root can edit both.</li><li>Timestamps and sizes are trivially restorable. A digest change with an unchanged size and timestamp is a specific tell that someone tried to cover their tracks.</li></ul>



<p class="wp-block-paragraph">Treat findings as strong evidence and a clean result as weak evidence. That asymmetry is the useful part.</p>



<h3 class="wp-block-heading">Application-level checksums</h3>



<p class="wp-block-paragraph">For anything running WordPress, WP-CLI covers the gap the package manager leaves, because it verifies against checksums published by wordpress.org rather than anything stored locally:</p>



<pre class="wp-block-code"><code>wp core verify-checksums
wp plugin verify-checksums --all</code></pre>



<p class="wp-block-paragraph">Neither command covers themes or premium plugins that aren&#8217;t in the public repository, so pair them with a modification-time sweep of the document root.</p>



<h3 class="wp-block-heading">Rootkit scanners</h3>



<p class="wp-block-paragraph">The honest case for <code>rkhunter</code> and <code>chkrootkit</code>: they&#8217;re free, they run in a minute, and they catch known families and common mistakes that an experienced pair of eyes might skim past. Running one costs you nothing.</p>



<p class="wp-block-paragraph">The limit: they&#8217;re signature and heuristic based, they check for kits that were known when the signature set was written, and they ask the same possibly compromised kernel and libc that everything else asks. A clean scan means the obvious things aren&#8217;t there. It is not a negative result.</p>



<h2 class="wp-block-heading">Failure family 4: the way in is still open</h2>



<p class="wp-block-paragraph">If you finish the cleanup without an answer to &#8220;how did they get in&#8221;, you haven&#8217;t finished. You&#8217;ve just reset the clock.</p>



<p class="wp-block-paragraph">The practical technique is to find the earliest artifact you&#8217;re confident about, then read the access logs around that timestamp. Attackers rarely land quietly on the first attempt, and the noise before the successful request is usually more legible than the request itself.</p>



<pre class="wp-block-code"><code># Files in the document root modified in the last two weeks
find /var/www -xdev -type f -newermt '14 days ago' -printf '%T+ %pn' 2&gt;/dev/null | sort

# Same idea for the usual drop locations
find /tmp /var/tmp /dev/shm -xdev -type f -newermt '14 days ago' -ls 2&gt;/dev/null</code></pre>



<p class="wp-block-paragraph">The usual suspects, roughly in the order I check them: an outdated plugin or CMS component with a public exploit, an admin panel exposed to the whole internet, a weak or reused password on an account with a shell, a deploy key or CI token leaked into a repository, and a service bound to <code>0.0.0.0</code> that was only ever meant to be local. Redis and an unauthenticated Docker socket are the two that come up most in that last category.</p>



<p class="wp-block-paragraph">If your logs for the relevant window are gone, that absence is itself a finding. Shipping logs off-box to somewhere append-only, whether that&#8217;s a hosted platform or your own Loki instance, is the single change that would have made this investigation tractable. Worth noting for the rebuild.</p>



<h2 class="wp-block-heading">The in-place technique that actually holds up</h2>



<p class="wp-block-paragraph">Everything above shares a weakness: you&#8217;re asking a possibly compromised system to describe itself. There&#8217;s one way around that which doesn&#8217;t require a reinstall, and it&#8217;s the reason I don&#8217;t dismiss in-place work entirely.</p>



<p class="wp-block-paragraph">Boot the VPS into your provider&#8217;s rescue image, or attach a snapshot of the disk to a second clean instance. Then mount the target filesystem read-only and inspect it from a kernel the attacker never touched.</p>



<pre class="wp-block-code"><code># In the rescue environment: identify the target disk
lsblk -f

# Mount read-only. On ext4 add noload, on XFS add norecovery,
# so the kernel doesn't replay the journal and write to the disk.
mkdir -p /mnt/target
mount -o ro,noload /dev/vdb1 /mnt/target      # ext4
# mount -o ro,norecovery /dev/vdb1 /mnt/target  # xfs

# Inspect without chrooting into it
ls -la /mnt/target/etc/cron.d/
cat /mnt/target/etc/ld.so.preload 2&gt;/dev/null
find /mnt/target/etc/systemd/system -newermt '30 days ago' -ls</code></pre>



<p class="wp-block-paragraph">Why this is different in kind, not just degree: a userspace rootkit hooking libc, or a kernel module hiding directory entries, works by intercepting calls on the running system. Mounted from outside, those files are just bytes. Nothing gets to filter what you see.</p>



<p class="wp-block-paragraph">Two things to keep straight. Don&#8217;t <code>chroot</code> into the mounted filesystem and start running its binaries, because you&#8217;ve just handed control back to the attacker&#8217;s code. And read-only isn&#8217;t automatic: mounting a dirty ext4 or XFS filesystem will replay the journal and write to the device unless you pass the option that suppresses it, which is enough to invalidate the copy if anyone later cares about chain of custody.</p>



<h2 class="wp-block-heading">When your checks disagree</h2>



<ul class="wp-block-list"><li><strong>Package verification is clean but the behaviour continues.</strong> The payload isn&#8217;t in a packaged file. Look at the document root, <code>/opt</code>, <code>/usr/local</code>, and anything in a home directory. That&#8217;s most of the filesystem, and none of it is covered.</li><li><strong>A process has no matching binary on disk.</strong> The executable was unlinked after launch. Check <code>ls -l /proc/PID/exe</code> for the <code>(deleted)</code> suffix and the original path, then find what restarts it, because something must.</li><li><strong>A file refuses to be deleted as root.</strong> Check <code>lsattr</code> for the immutable attribute and clear it with <code>chattr -i</code>. Attackers use this specifically because it makes casual cleanup fail silently.</li><li><strong>Rebooting appears to fix it and it comes back on a schedule.</strong> Enumerate every timer and cron path again, including <code>systemctl list-timers --all</code> and per-user units under lingering sessions.</li><li><strong>Journal entries are missing for the interesting window.</strong> Run <code>journalctl --verify</code>. Truncation is a finding, not a dead end, and it argues for moving to the offline path.</li><li><strong>The outbound traffic continues after you&#8217;ve killed everything.</strong> You&#8217;re either looking at a hidden process or a second compromised container on the same host. Confirm from the provider&#8217;s traffic graphs rather than from tools on the box.</li></ul>



<h2 class="wp-block-heading">Arguments for staying in place that don&#8217;t survive contact</h2>



<ul class="wp-block-list"><li><strong>&#8220;The rootkit scanner came back clean.&#8221;</strong> It checked for known kits using tools running on the suspect system. Absence of evidence, and weak absence at that.</li><li><strong>&#8220;I rotated all the passwords.&#8221;</strong> Passwords are one access path. SSH keys, API tokens, sudo rules, PAM modules and scheduled jobs are the others, and none of them care about your password reset.</li><li><strong>&#8220;It&#8217;s only a web shell, the OS is fine.&#8221;</strong> Sometimes true, and it&#8217;s the one case where in-place recovery is genuinely defensible. But it&#8217;s a conclusion you have to earn by ruling out privilege escalation, not an assumption you start from.</li><li><strong>&#8220;I restored from backup.&#8221;</strong> Restoring content doesn&#8217;t remove a key from <code>/root/.ssh</code>, and if the backup predates your detection by less than the dwell time, you&#8217;ve restored the compromise. Know when the earliest artifact appeared before you pick a restore point.</li><li><strong>&#8220;I don&#8217;t have time to rebuild.&#8221;</strong> The honest version of this is a real constraint, not a technical argument. But if your infrastructure is described in code and your data restore is tested, a rebuild is often faster than the second cleanup, never mind the third.</li></ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<ol class="wp-block-list"><li><strong>Snapshot first, always.</strong> Before any judgement call. It&#8217;s cheap and it&#8217;s the only irreversible decision in the whole process.</li><li><strong>Isolate at the provider edge</strong> and collect volatile state while the box is still up.</li><li><strong>Establish depth.</strong> Did anything run as root? If you can&#8217;t answer that with evidence, assume yes.</li><li><strong>If root is on the table, plan the rebuild now</strong> and treat the in-place work as investigation rather than remediation. The two goals stop competing once you&#8217;ve decided which one you&#8217;re pursuing.</li><li><strong>If it stayed inside one unprivileged account,</strong> clean in place: remove the payload, close the access paths, patch the entry point, rotate every credential that account could read.</li><li><strong>Do the offline read-only pass anyway.</strong> It&#8217;s twenty minutes in a rescue image and it&#8217;s the only evidence that doesn&#8217;t depend on the compromised machine&#8217;s cooperation.</li><li><strong>Rebuild the hardening on the new instance,</strong> not the old one. Key-only SSH, an allowlist on the admin port, automatic security updates, off-box logging, and file integrity monitoring with AIDE or a platform like Wazuh, so the next incident starts with a baseline instead of guesswork.</li></ol>



<p class="wp-block-paragraph">On that allowlist point: restricting SSH to a fixed source address is the single highest-value control here, and it&#8217;s the one people skip because their own IP changes. A VPN with a dedicated IP, which providers like NordVPN and Surfshark both offer, or a Cloudflare Tunnel in front of the admin path, gives you a stable origin to allow without exposing the port to the internet.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can you ever recover a hacked Linux server without reinstalling?</h3>



<p class="wp-block-paragraph">Yes, when the compromise stayed inside an unprivileged application account and you can demonstrate that it did. Once anything ran as root, you can restore service in place but you cannot honestly certify the machine is clean, because every tool you&#8217;d use to check runs under the code you&#8217;re checking for.</p>



<h3 class="wp-block-heading">Should I reboot the VPS as soon as I notice something wrong?</h3>



<p class="wp-block-paragraph">Not first. A reboot destroys the process list, open sockets, and the mapping from running processes to deleted binaries, which is often the most informative evidence available. Snapshot, isolate at the edge, collect volatile state, then reboot if you need to.</p>



<h3 class="wp-block-heading">How long do attackers usually stay before I notice?</h3>



<p class="wp-block-paragraph">Long enough that you should be suspicious of any restore point close to your detection date. Rather than trusting a published average, find the earliest artifact you&#8217;re confident about on your own machine, then add margin. If your logs don&#8217;t reach back that far, that&#8217;s a finding in itself.</p>



<h3 class="wp-block-heading">Is restoring from a backup enough after a compromise?</h3>



<p class="wp-block-paragraph">Only if the backup predates the intrusion and covers everything that was modified. A content-only restore leaves SSH keys, cron entries, systemd units and package changes in place, so it usually removes the visible damage while preserving the access. Restore data onto a fresh instance instead of over a compromised one.</p>



<h3 class="wp-block-heading">What&#8217;s the fastest way to check for a compromised VPS?</h3>



<p class="wp-block-paragraph">Sockets with owning processes (<code>ss -tunap</code>), authorized keys across every home directory, every timer including inactive ones (<code>systemctl list-timers --all</code>), <code>/etc/ld.so.preload</code>, and package verification. That&#8217;s a few minutes and it catches the majority of commodity intrusions. It will not catch a competent one.</p>



<h3 class="wp-block-heading">Do I need to tell my hosting provider?</h3>



<p class="wp-block-paragraph">If they contacted you about abuse traffic, yes, and quickly, because unanswered abuse reports lead to suspension. It&#8217;s also worth asking whether they can supply traffic graphs for the affected period. That data comes from outside your instance, which makes it more trustworthy than anything on it.</p>



<h3 class="wp-block-heading">If there was personal data on the server, does that change anything?</h3>



<p class="wp-block-paragraph">It changes the order of operations. Preserve the snapshot before you clean anything, document what you did and when, and check your notification obligations before the technical work absorbs your attention. Cleanup that destroys evidence is a much bigger problem when the incident is reportable.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Compromised VPS recovery isn&#8217;t really a cleanup procedure. It&#8217;s a decision about what you&#8217;re able to claim afterwards.</p>



<p class="wp-block-paragraph">If the intrusion stayed inside an unprivileged account and you can show it, clean in place and move on. If root was ever involved, every check you run on that box is a statement from a machine with a motive. Take the snapshot, do the offline read-only pass because it&#8217;s the only view the attacker doesn&#8217;t control, and put the rebuilt hardening on a fresh instance rather than the one you&#8217;ve been arguing with.</p>



<p class="wp-block-paragraph">And find the way in. A cleanup that doesn&#8217;t end with a patched entry point isn&#8217;t a recovery, it&#8217;s an interval.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on a compromised server?</h2>



<p class="wp-block-paragraph">Incident work is unpleasant to do alone and easy to do in the wrong order. This is the kind of thing I help with:</p>



<ul class="wp-block-list"><li>Triage on a live VPS: establishing depth of compromise, collecting volatile evidence before it&#8217;s lost, and giving you a straight answer on clean-in-place versus rebuild</li><li>Offline analysis of a snapshot or rescue-mode disk, including timeline reconstruction from filesystem timestamps and whatever logs survived</li><li>Full enumeration of access paths and persistence: SSH configuration, PAM, sudoers, cron, systemd units and timers, preload hooks and setuid binaries</li><li>Finding the entry point rather than just the payload, correlating web and auth logs against the earliest artifacts on disk</li><li>Rebuilding the server properly afterwards, in code, with key-only SSH, an admin allowlist, off-box logging and file integrity monitoring in place from day one</li><li>WordPress and PHP application cleanup where the compromise came in through the stack rather than the OS</li></ul>



<p class="wp-block-paragraph">If you&#8217;re in the middle of one right now, send me the output of <code>ss -tunap</code>, your <code>systemctl list-timers --all</code>, and the abuse report if you got one. That&#8217;s usually enough to tell you how deep this goes.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/system-administration/compromised-vps-recovery-without-reinstall/">Compromised VPS Recovery: What You Can Actually Fix Without Reinstalling</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Designing a Construction Dispute Intelligence Platform That Survives Cross-Examination</title>
		<link>https://john-nessime.com/blog/technical-guides/construction-dispute-intelligence-platform/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 20:19:52 +0000</pubDate>
				<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[As-Built Schedule]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[Bates Numbering]]></category>
		<category><![CDATA[Capital Projects]]></category>
		<category><![CDATA[Chain of Custody]]></category>
		<category><![CDATA[Common Data Environment]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Delay Analysis]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Embeddings]]></category>
		<category><![CDATA[Extension of Time]]></category>
		<category><![CDATA[Forensic Schedule Analysis]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Hybrid Search]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[Load Files]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Oracle Aconex]]></category>
		<category><![CDATA[pgvector]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[Project Controls]]></category>
		<category><![CDATA[Provenance]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Reciprocal Rank Fusion]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[System of Record]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<category><![CDATA[XER Files]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=336</guid>

					<description><![CDATA[<p>A layer-by-layer architecture for a construction dispute intelligence platform: hashed intake and immutable originals, OCR with page coordinates, a versioned Primavera P6 schedule plane, an event model that joins documents to activities, hybrid retrieval on pgvector, and a generation layer bounded by human confirmation. Written around what each layer prevents.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/construction-dispute-intelligence-platform/">Designing a Construction Dispute Intelligence Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that kills these systems is always the same one, and it always comes from the expert, not the engineer: &#8220;Where did this date come from?&#8221;</p>



<p class="wp-block-paragraph">You have a screen showing a beautiful chronology. Two hundred events, colour-coded by discipline, cross-referenced against the programme. The delay expert points at row 47, the one that says the piling rig arrived late, and asks which document that came from, who produced it, what page, and whether the other side has seen it. If the answer involves anyone opening a folder and searching, the platform has already failed. Not technically. Evidentially. A chronology nobody can trace back to a produced document is a nice picture, and a nice picture is worth nothing in an adjudication.</p>



<p class="wp-block-paragraph">That is the constraint that shapes everything else. A construction dispute intelligence platform is not a search engine with a construction theme on it. It is a provenance system that happens to do search. Get that ordering wrong and you build something impressive that no expert will sign their name to.</p>



<p class="wp-block-paragraph">This post walks through the architecture layer by layer: intake and chain of custody, normalization, the schedule data plane, the chronology model that joins the two, retrieval, and the tightly bounded generation layer on top. For each one I&#8217;ll say what it prevents, because that is the only reason any of these layers exist.</p>



<h2 class="wp-block-heading">What a construction dispute intelligence platform actually has to do</h2>



<p class="wp-block-paragraph">Strip away the vocabulary and the job is narrow. Take a pile of heterogeneous records from a capital project, some of them structured and some of them scanned faxes, and let a small number of expensive humans answer three questions quickly and defensibly:</p>



<ul class="wp-block-list">
<li>What happened, in what order, and who said so at the time?</li>



<li>What did the programme say before and after each of those things happened?</li>



<li>Which contemporaneous record supports every single assertion in the report?</li>
</ul>



<p class="wp-block-paragraph">The third one is not a nice-to-have. Both of the industry references people actually argue from lean on it. The Society of Construction Law Delay and Disruption Protocol raised record keeping to a core principle in its second edition and stopped expressing a preference between retrospective delay analysis methods, on the reasoning that the records you have largely determine the method available to you. AACE International&#8217;s Recommended Practice 29R-03 codifies forensic schedule analysis into a taxonomy of Method Implementation Protocols, and its minimum implementation protocols are essentially statements about which source schedules you must be able to produce. Both documents are, underneath, about evidence handling. So is your architecture.</p>



<h2 class="wp-block-heading">Layer one: intake that a forensic expert can defend</h2>



<p class="wp-block-paragraph">Everything upstream of this layer is somebody else&#8217;s mess. Site photos on a phone, a shared drive nobody pruned since mobilization, an Aconex export, a hard drive that arrived in an envelope. Your job at intake is to make an irreversible statement about what you received and when, before anyone touches it.</p>



<p class="wp-block-paragraph">Hash every file on arrival and store the hash next to the object, not in a spreadsheet. SHA-256 over the raw bytes, computed before any conversion, deduplication or renaming. The hash becomes the document&#8217;s identity for the rest of its life.</p>



<pre class="wp-block-code"><code># Build an arrival manifest before anything else runs
find ./ACME-INTAKE-001 -type f -print0 
  | xargs -0 sha256sum &gt; manifest-ACME-INTAKE-001.txt

# Re-verify later, after the files have moved through processing
sha256sum -c manifest-ACME-INTAKE-001.txt</code></pre>



<p class="wp-block-paragraph">Then put the originals somewhere they cannot be quietly edited. On AWS, S3 Object Lock in compliance mode with a retention period is the blunt instrument: once set, the object version cannot be overwritten or deleted by anyone, including the root account, until retention expires. Governance mode is the softer variant, where a specifically permissioned principal can lift the lock, which is fine for internal working copies and wrong for originals. If you are self-hosting on a Contabo or InterServer box because the client will not put project data in a hyperscaler, the equivalent discipline is a write-once staging volume plus an append-only manifest log, and you accept that you are relying on process rather than on the storage layer refusing the write.</p>



<h3 class="wp-block-heading">The manifest is the product</h3>



<p class="wp-block-paragraph">If the matter is already in formal disclosure, you will be handed productions rather than raw folders, and those arrive as load files. A Concordance DAT carries one row per document with fields like BegBates, EndBates, Custodian, BegAttach, EndAttach and an MD5 hash. An Opticon OPT carries one row per page, mapping Bates numbers to image paths. The delimiters are deliberately weird characters that real text rarely contains, and different tools disagree about the exact code points, so parse defensively and log what you detected rather than assuming.</p>



<p class="wp-block-paragraph">The failure mode here is quiet and expensive. Variable-width Bates numbers sort lexically instead of numerically, so ABC10 lands between ABC1 and ABC2, attachment ranges resolve to the wrong parents, and a family of documents silently splits. Nobody notices until an expert cites a covering email whose attachment, in your system, belongs to a different letter. Pad to fixed width on ingest and reconcile every BegAttach range against the documents you actually loaded.</p>



<h2 class="wp-block-heading">Layer two: normalization, and the OCR tax nobody budgets for</h2>



<p class="wp-block-paragraph">Construction corpora are the worst kind of mixed. Native Office files with clean text, born-digital PDFs, scanned site instructions with a signature and a coffee ring, photographs of whiteboards, and drawings where the only text that matters is inside a title block in the bottom right corner.</p>



<p class="wp-block-paragraph">Route by what the file actually is, not by extension. For a PDF, try text extraction first; if the extracted layer is empty or nearly empty, it is a scan and needs OCR. Tesseract via OCRmyPDF is the cheap path and it is genuinely good on clean typed pages. It degrades badly on handwriting, on stamps, and on anything skewed, which describes most site paperwork. A managed service such as Amazon Textract handles tables and forms better and returns geometry for every detected block, which matters more than accuracy alone.</p>



<p class="wp-block-paragraph">Geometry is the part teams skip and then rebuild six months later. If you store only the extracted string, you can prove a phrase exists somewhere in a document. If you store the page number and the bounding box, you can put a highlight on the exact line in the viewer, and the expert can confirm it in two seconds instead of reading eleven pages. That difference is the difference between a tool people use and a tool people abandon.</p>



<p class="wp-block-paragraph">Keep the OCR text separate from the original in storage, and record which engine and which settings produced it. When a disputed reading of a handwritten date turns up, and it will, you want to be able to say exactly how that character was produced.</p>



<h2 class="wp-block-heading">Layer three: the schedule plane is a separate data model</h2>



<p class="wp-block-paragraph">This is where most teams underestimate the work. Programme data is not documents. It is a graph, versioned, with a calendar attached, and it needs its own tables, its own ingest path and its own validation.</p>



<p class="wp-block-paragraph">Primavera P6 exports as XER, a tab-delimited text format with a table-per-section layout, or as P6 XML. Either way you are pulling out projects, the WBS tree, activities, logic links and calendars. In XER these live in sections including PROJECT, PROJWBS, TASK, TASKPRED and CALENDAR. Parsing it by hand is a bad use of a week; PyP6Xer is an open source Python parser that will get you to objects quickly.</p>



<pre class="wp-block-code"><code>from xerparser.reader import Reader

xer = Reader("update-14.xer")

for project in xer.projects:
    print(project.proj_short_name)
    for activity in project.activities:
        print(activity.task_code, activity.task_name)</code></pre>



<p class="wp-block-paragraph">Two things bite immediately. First, several duration and float fields in XER are stored in hours rather than days, which you can see in the column names themselves. If you load them straight into a &#8220;days&#8221; column because the numbers looked plausible on a five-day calendar, every float figure downstream is wrong by a factor that varies with the calendar. Check your own export before you trust any mapping, including this one.</p>



<p class="wp-block-paragraph">Second, activity IDs are not stable across a project&#8217;s life. Activities get renumbered, split, deleted and re-added between updates. If your schema keys on the activity ID alone, two updates will silently merge activities that were never the same work. Key on a composite of schedule version plus activity ID, and maintain a separate mapping table for identity across versions that a human can correct. That table is boring, it is not automatable, and it is the difference between a windows analysis that holds together and one that quietly compares apples to a different orchard.</p>



<p class="wp-block-paragraph">Store every update as an immutable snapshot with its data date. Never mutate a loaded schedule. The whole point of the layer is that you can reconstruct exactly what the programme said on a given day, which is precisely what a time slice or windows method needs.</p>



<h2 class="wp-block-heading">Layer four: the chronology is the join, and the join is the hard part</h2>



<p class="wp-block-paragraph">Documents on one side, schedule snapshots on the other. The product is the thing in the middle: an event, with a date, a description, a set of supporting document references, and a set of affected activities.</p>



<p class="wp-block-paragraph">Model it explicitly. An event is not a document and it is not an activity. One letter can create three events. One event can be evidenced by nine documents and a photograph. The relationship is many-to-many in both directions and every attempt I have seen to flatten it ends with someone maintaining a spreadsheet on the side.</p>



<pre class="wp-block-code"><code>-- Every assertion resolves to a page and a box on that page
CREATE TABLE evidence_ref (
  id            BIGSERIAL PRIMARY KEY,
  event_id      BIGINT NOT NULL REFERENCES event(id),
  document_id   UUID   NOT NULL REFERENCES document(id),
  page_number   INT    NOT NULL,
  bbox          JSONB,             -- normalized coordinates from OCR
  quoted_text   TEXT   NOT NULL,
  asserted_by   TEXT   NOT NULL,   -- 'analyst' or 'model'
  confirmed_by  TEXT,              -- NULL until a human confirms
  confirmed_at  TIMESTAMPTZ
);</code></pre>



<p class="wp-block-paragraph">Those last three columns are the ones that matter. They encode the difference between something the machine proposed and something a person stands behind. Report generation reads only rows where <code>confirmed_by</code> is set. Everything else is a suggestion queue. This is not a compliance box; it is the mechanism by which an expert can put their name on the output, and it needs to exist in the schema rather than in a workflow convention that erodes under deadline.</p>



<h2 class="wp-block-heading">Layer five: retrieval that finds document numbers, not just meaning</h2>



<p class="wp-block-paragraph">Pure vector search is the wrong default for this corpus, and it fails in a way that looks like success. Embeddings are good at meaning and bad at tokens. Ask for &#8220;RFI 0442&#8221; and a dense retriever will cheerfully hand you eleven chunks about requests for information, none of which is RFI 0442, because the identifier got smeared into its semantic neighbourhood. Construction disputes run on identifiers: RFI numbers, variation order references, drawing revisions, activity IDs, Bates ranges.</p>



<p class="wp-block-paragraph">Run lexical and semantic retrieval in parallel and fuse the rankings. PostgreSQL with the pgvector extension does both without adding infrastructure: an HNSW index over the embedding column, a GIN index over a generated tsvector column, and reciprocal rank fusion to merge the two candidate lists.</p>



<pre class="wp-block-code"><code>CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunk (
  id          BIGSERIAL PRIMARY KEY,
  document_id UUID NOT NULL,
  page_number INT  NOT NULL,
  body        TEXT NOT NULL,
  body_tsv    tsvector GENERATED ALWAYS AS
                (to_tsvector('english', body)) STORED,
  embedding   vector(1536)
);

CREATE INDEX chunk_tsv_idx  ON chunk USING gin (body_tsv);
CREATE INDEX chunk_hnsw_idx ON chunk USING hnsw (embedding vector_cosine_ops);</code></pre>



<p class="wp-block-paragraph">Over-fetch from each side and trim after fusion rather than pulling the final count from each; more candidates give the fusion step something to work with. Add a trigram index if your corpus has OCR noise, because &#8220;RFI 0442&#8221; and &#8220;RFl 0442&#8221; are the same document to a human and different tokens to Postgres.</p>



<p class="wp-block-paragraph">Filter hard on metadata before ranking. Matter, custodian, date range, privilege status. A retrieval layer that can return a privileged document into a shared workspace is a liability, not a feature.</p>



<h2 class="wp-block-heading">Layer six: generation, bounded hard</h2>



<p class="wp-block-paragraph">Language models are useful here and the useful jobs are smaller than the demos suggest: proposing candidate events from a batch of correspondence, drafting neutral one-line summaries, clustering near-duplicate records, suggesting which activities an event might touch. Every one of those is a proposal that lands in a review queue.</p>



<p class="wp-block-paragraph">The reason to be strict is not squeamishness. Stanford&#8217;s RegLab and HAI ran a preregistered evaluation of commercial legal research tools built on retrieval-augmented generation, with expert-scored answers, and found hallucination rates in the range of roughly 17 to 33 percent depending on the product, against vendor marketing that had claimed hallucination-free citations. The vendors disputed the characterization. The finding that should worry you more than the headline number is the failure mode underneath it: a citation that exists, is quoted accurately, and does not stand for what the answer says it stands for. Retrieval reduces fabrication. It does not produce correct reasoning about what a document means, and misgrounding is far harder to spot than an invented reference.</p>



<p class="wp-block-paragraph">So bound it. Generation writes to the proposal table, never to the report. Every generated assertion carries the chunk IDs it was built from, and the UI shows the source page beside the claim by default rather than behind a click. If a proposed event has no evidence reference, it does not enter the queue at all.</p>



<h2 class="wp-block-heading">Tenancy, privilege, and the thing that ends the engagement</h2>



<p class="wp-block-paragraph">Matters do not share data. Ever. Not &#8220;logically separated by a WHERE clause that we always remember to add&#8221;. The cheapest defensible pattern is a database per matter, or at minimum a schema per matter with connection-level scoping, so that a missing predicate produces an empty result rather than another client&#8217;s correspondence.</p>



<p class="wp-block-paragraph">Two more that get discovered late. Privilege state is per document per matter and it changes; model it as a versioned attribute with an audit trail, not a boolean someone flipped. And plan the exit at the start: matters end, retention clocks expire, and someone has to certify destruction. That means knowing every copy you made, including OCR text, thumbnails, embeddings and backups. Tools like the O&amp;O Software utilities have a place at the endpoint end of that job; the platform side is a deletion manifest that enumerates derived artifacts per document, written at ingest rather than reconstructed under pressure.</p>



<p class="wp-block-paragraph">If experts are working from home, from site cabins, or from three countries, the access path is part of the design. A managed VPN such as NordVPN or Surfshark on the client side plus network-level allowlisting on the platform side is a pragmatic combination, and it is worth writing into the engagement documentation so nobody improvises with a personal Dropbox at eleven at night.</p>



<h2 class="wp-block-heading">Where these platforms break in practice</h2>



<ul class="wp-block-list">
<li><strong>Chronology dates drift from document dates.</strong> Usually a timezone or a date-format collision: a scanned form read as DD/MM, an email header parsed as UTC, a schedule data date stored as a naive timestamp. Store the raw string you extracted alongside the parsed value and show both in the review UI.</li>



<li><strong>Float figures look wrong by a constant factor.</strong> Hours versus days in the schedule import. Check against the same activity opened in P6 before you debug anything else.</li>



<li><strong>Retrieval misses a document everyone knows exists.</strong> Either it was never OCR&#8217;d because the PDF had a thin text layer that passed your emptiness check, or the identifier is only in the title block of a drawing. Log extraction character counts per page and alert on suspiciously low ones.</li>



<li><strong>Two activities merged across updates.</strong> Composite keying was not applied, or an activity was renumbered. Diff activity code sets between consecutive snapshots and surface additions and deletions as a report the planner reviews.</li>



<li><strong>Document families split.</strong> Bates padding or an unreconciled attachment range from the load file. Re-run the family reconciliation and count orphaned attachments.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the schedule as documents. It is a graph with versions, and shoehorning it into the document store costs you every analysis you actually wanted.</li>



<li>Storing extracted text without page and coordinate provenance. Cheap now, a full reprocessing run later.</li>



<li>Vector-only retrieval. Looks great in a demo built from prose, falls over on the first RFI number.</li>



<li>Letting model output reach a deliverable without a recorded human confirmation.</li>



<li>Mutating loaded schedules to &#8220;fix&#8221; them. Fix on the way in, or hold the correction as an annotation. Never edit the snapshot.</li>



<li>Deferring the deletion story to the end of the matter, when the derived artifacts are already scattered.</li>
</ul>



<h2 class="wp-block-heading">Best practices for building a construction dispute intelligence platform</h2>



<ul class="wp-block-list">
<li>Hash at the door, lock the originals, and never process in place.</li>



<li>Make every derived artifact point back to a document ID, a page and a region.</li>



<li>Give schedules their own immutable, version-keyed store with the data date as a first-class field.</li>



<li>Model events separately from both documents and activities, with many-to-many links in each direction.</li>



<li>Run hybrid retrieval and fuse the rankings; filter on matter and privilege before you rank.</li>



<li>Keep generation in a proposal queue with confirmation recorded in the schema.</li>



<li>Isolate matters at the database or schema boundary, not in application code.</li>



<li>Instrument ingest as seriously as you would a production pipeline. Grafana over your extraction counts and OCR queue depth will tell you a corpus is going wrong days before an analyst does.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a dedicated vector database for this?</h3>



<p class="wp-block-paragraph">Almost certainly not. Matter corpora are large in gigabytes and small in chunk count compared to consumer-scale search. PostgreSQL with pgvector keeps the embeddings in the same transaction as the provenance rows, which means you cannot end up with a chunk whose document record was rolled back. That consistency is worth more here than the last few milliseconds of query latency.</p>



<h3 class="wp-block-heading">Should the platform pick a delay analysis method?</h3>



<p class="wp-block-paragraph">No. That choice belongs to the expert and depends on the contract, the forum, the records available and the timing of the analysis. What the platform should do is make it obvious which methods the record set can support, by reporting what you actually have: how many schedule updates, at what intervals, with what data dates, and how complete the contemporaneous correspondence is around each window.</p>



<h3 class="wp-block-heading">XER or P6 XML for schedule ingest?</h3>



<p class="wp-block-paragraph">Accept both, because you will be sent both. XER is what planners export by habit and it is easy to parse with existing libraries. P6 XML is more explicit about structure and less prone to encoding surprises. Whichever arrives, keep the original file, hashed, alongside the parsed rows.</p>



<h3 class="wp-block-heading">How much of the review can realistically be automated?</h3>



<p class="wp-block-paragraph">The ranking and the grouping, most of it. The judgement, none of it. Automation that reduces a hundred thousand documents to a few thousand worth reading is enormously valuable and uncontroversial. Automation that decides what a letter means is where you inherit risk you cannot price.</p>



<h3 class="wp-block-heading">Can I build this on top of a common data environment we already have?</h3>



<p class="wp-block-paragraph">Partly. A CDE such as Aconex is a good source of records and a poor system of record for a dispute, because access changes, exports are point-in-time, and the retention rules belong to the project rather than to the matter. Pull from it, hash what you pulled, and hold your own copy under your own lock.</p>



<h3 class="wp-block-heading">What does this cost to run?</h3>



<p class="wp-block-paragraph">The shape of the bill is more useful than a number. OCR and embedding are one-time per document and scale with page count, so a scan-heavy corpus costs multiples of a born-digital one. Storage is small and constant. Retrieval is negligible. Generation scales with how much you push through it, which is another argument for narrow, queued use rather than a chat box over the whole corpus.</p>



<h3 class="wp-block-heading">Is a general eDiscovery platform not enough?</h3>



<p class="wp-block-paragraph">For document review, often yes, and they are considerably better at productions and privilege logging than anything you will build in a quarter. What they do not have is the schedule plane. The value of a construction-specific build is the join between correspondence and programme versions, so a sensible architecture frequently uses an established review platform for documents and adds the schedule and chronology layers alongside it.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A construction dispute intelligence platform earns its place by making provenance cheap. Search, clustering and summarization are commodity capabilities now; anyone can bolt them onto a corpus in a fortnight. What is hard, and what decides whether an expert will rely on the output, is that every date, every event and every sentence in the eventual report resolves in one click to a specific page of a specific produced document, with a record of who confirmed it.</p>



<p class="wp-block-paragraph">Build the provenance spine first. Hash at intake, keep coordinates, version the schedules, model events as their own thing, and keep the model on the proposal side of a human confirmation. The clever features are easy to add afterwards. Retrofitting evidence handling into a system that was designed as a search tool is a rebuild, and it always arrives at the worst possible moment, which is about three weeks before a hearing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or fixing this?</h2>



<p class="wp-block-paragraph">I work on the infrastructure side of exactly this problem: getting messy project records into a system where every claim is traceable. Things I can help with:</p>



<ul class="wp-block-list">
<li>Designing the intake and chain-of-custody layer, including hashing, immutable storage, and load file parsing that survives odd delimiters and inconsistent Bates padding.</li>



<li>Building the schedule ingest path for XER and P6 XML, with version-keyed snapshots and an activity identity map that holds across updates.</li>



<li>Standing up hybrid retrieval on PostgreSQL and pgvector, including page-level and coordinate-level provenance so highlights land on the right line.</li>



<li>Setting up the OCR and normalization pipeline, with routing rules, quality checks, and alerting on pages that extracted suspiciously little text.</li>



<li>Matter isolation, privilege state modelling, audit logging, and a deletion manifest you can act on at the end of the engagement.</li>



<li>Reviewing an existing platform and telling you plainly where the provenance chain breaks.</li>
</ul>



<p class="wp-block-paragraph">If you have something concrete, send it over: an XER export, a sample DAT and OPT pair, a page count breakdown of your corpus, or the schema you are currently working from. It is much easier to say something useful about a real file than about a description of one.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/construction-dispute-intelligence-platform/">Designing a Construction Dispute Intelligence Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
