<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Construction Technology | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/construction-technology/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/construction-technology/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 25 Oct 2026 06:06:42 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Construction Technology | John Nessime</title>
	<link>https://john-nessime.com/blog/construction-technology/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>The Delay Was in the Data Three Weeks Ago: Building a Construction Early Warning Dashboard</title>
		<link>https://john-nessime.com/blog/data-engineering/construction-early-warning-dashboard/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 25 Oct 2026 06:06:42 +0000</pubDate>
				<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Capital Projects]]></category>
		<category><![CDATA[Delay Analysis]]></category>
		<category><![CDATA[Earned Schedule]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Last Planner System]]></category>
		<category><![CDATA[Leading Indicators]]></category>
		<category><![CDATA[Owner Dashboard]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[Procore]]></category>
		<category><![CDATA[Project Controls]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Total Float]]></category>
		<category><![CDATA[XER Files]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=1033</guid>

					<description><![CDATA[<p>Schedule delay rarely arrives as a surprise. It sits in float erosion, aging RFIs and slipping weekly commitments long before the monthly update moves the finish date. Here is how to build a dashboard that catches it while there is still float left to spend.</p>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/construction-early-warning-dashboard/">The Delay Was in the Data Three Weeks Ago: Building a Construction Early Warning Dashboard</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The monthly schedule update lands on a Thursday. The planner has re-run the logic, and substantial completion has moved three weeks to the right. The project manager asks the obvious question in the next meeting: why is this the first time anyone is hearing about it?</p>



<p class="wp-block-paragraph">The uncomfortable answer is that people did know. The superintendent knew the drywall crew had been short for two weeks. The project engineer knew a structural RFI had been sitting with the design team well past its due date. The foreman knew the weekly commitments kept slipping. None of it was hidden. It just never reached one screen, in one place, while there was still float left to spend.</p>



<p class="wp-block-paragraph">This post walks through building a <strong>construction early warning dashboard</strong> that catches those signals while they are still cheap to act on. It covers why schedule delay shows up late, the one data decision that makes early warning possible, the signals worth tracking, how to pull data out of Primavera P6, Microsoft Project and Procore, and how to alert without training everyone to ignore the alerts.</p>



<h2 class="wp-block-heading">Why Construction Delay Shows Up Three Weeks Late</h2>



<p class="wp-block-paragraph">Delay rarely arrives as one event. It accumulates in small amounts across activities that do not look critical yet. Three things keep it invisible.</p>



<ol class="wp-block-list">
<li><strong>The schedule is updated in batches.</strong> If the CPM schedule is statused monthly, or even biweekly, the finish date is only recalculated at that cadence. Everything that happened in between lands at once.</li>



<li><strong>Progress is self-reported.</strong> Percent complete is an estimate from whoever statuses the activity. Optimism is not dishonesty, but it compounds. An activity that sits at a comfortable-looking percentage for three updates is telling you something.</li>



<li><strong>The data is overwritten in place.</strong> This is the one that bites hardest. Most schedule tools, and most dashboards built on top of them, show the current value of float, the current forecast finish and the current status. Last month&#8217;s values are gone unless someone saved the file.</li>
</ol>



<p class="wp-block-paragraph">That third point is why so many project dashboards look fine right up until they don&#8217;t. A current-state dashboard tells you where you are. Delay warnings live in the rate of change. An activity with twelve days of float looks healthy. The same activity with twelve days of float, down from thirty over the last two updates, is the next critical path.</p>



<h2 class="wp-block-heading">The Design Decision That Makes Early Warning Possible: Snapshot Everything</h2>



<p class="wp-block-paragraph">Before any chart or alert, decide that every schedule update, every RFI log pull and every weekly plan gets stored as a dated snapshot. You append. You never overwrite. Storage for this is trivial; a few hundred activities per update is nothing for any database.</p>



<p class="wp-block-paragraph">PostgreSQL is what I reach for first here. It runs happily on a small managed instance like Amazon RDS or DigitalOcean Managed Databases, or on a modest VPS from Hetzner, Contabo or Interserver if you are comfortable handling backups yourself. A minimal activity snapshot table looks like this:</p>



<pre class="wp-block-code"><code>CREATE TABLE activity_snapshot (
    project_code     text        NOT NULL,
    data_date        date        NOT NULL,
    task_code        text        NOT NULL,
    task_name        text,
    total_float_days numeric,
    early_start      date,
    early_finish     date,
    is_complete      boolean     NOT NULL DEFAULT false,
    loaded_at        timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (project_code, data_date, task_code)
);</code></pre>



<p class="wp-block-paragraph">The primary key does the important work. Each activity appears once per data date, so loading the same export twice fails loudly instead of silently doubling the history. The <code>data_date</code> is the schedule&#8217;s status date, not the day you loaded the file, because trends must be plotted against when the schedule says it was true.</p>



<p class="wp-block-paragraph">Keying on <code>task_code</code>, the activity ID the planner sees, rather than an internal database ID matters too. Internal IDs can change when a schedule is re-imported. Activity IDs usually survive, and when they don&#8217;t, you want to know (more on that in troubleshooting).</p>



<h2 class="wp-block-heading">Early Warning Signals That Move Before the Finish Date Does</h2>



<p class="wp-block-paragraph">Each of these is a leading indicator of schedule delay. None is perfect alone. Together they give you weeks of notice instead of a surprise in the monthly update.</p>



<h3 class="wp-block-heading">Float Erosion on Near-Critical Paths</h3>



<p class="wp-block-paragraph">Watching the critical path is necessary but late. By the time an activity is critical, its float is already gone. The useful signal is near-critical work losing float quickly. This query compares each open activity to its own previous snapshot:</p>



<pre class="wp-block-code"><code>WITH history AS (
    SELECT
        project_code,
        task_code,
        task_name,
        data_date,
        total_float_days,
        LAG(total_float_days) OVER (
            PARTITION BY project_code, task_code
            ORDER BY data_date
        ) AS prev_float_days
    FROM activity_snapshot
    WHERE NOT is_complete
)
SELECT
    project_code,
    task_code,
    task_name,
    prev_float_days,
    total_float_days,
    prev_float_days - total_float_days AS float_lost_days
FROM history
WHERE data_date = (SELECT max(data_date) FROM activity_snapshot)
  AND prev_float_days IS NOT NULL
  AND total_float_days &lt;= 20
  AND prev_float_days - total_float_days &gt;= 5
ORDER BY float_lost_days DESC;</code></pre>



<p class="wp-block-paragraph">The thresholds (twenty days remaining, five days lost) are starting points, not rules. Tune them to your update cadence and project length. What matters is that the query looks at movement, which a current-state report cannot.</p>



<h3 class="wp-block-heading">Constraint Aging: RFIs and Submittals Against Need-By Dates</h3>



<p class="wp-block-paragraph">An overdue RFI on its own is noise. Every project has them. An overdue RFI that blocks an activity starting in ten days is a delay you can see coming.</p>



<p class="wp-block-paragraph">The hard part is the link. RFI and submittal logs in Procore or Autodesk Construction Cloud rarely point at schedule activity IDs. Someone has to maintain a small mapping table from RFI or submittal to activity, or at least to area and spec section. I would not skip it. Without the link you get a list of late paperwork; with it you get &#8220;this answer is needed in eight days and the ball has been in the architect&#8217;s court for three weeks.&#8221;</p>



<p class="wp-block-paragraph">The signal to chart is days of slack between the expected response and the need-by date of the linked activity. When it goes negative, the RFI is now on the schedule&#8217;s path whether the schedule knows it or not.</p>



<h3 class="wp-block-heading">Commitment Reliability: PPC and Reasons for Variance</h3>



<p class="wp-block-paragraph">If the field runs the Last Planner System, you already have a strong signal. Percent Plan Complete (PPC) is the share of weekly commitments finished as promised. Each week&#8217;s PPC is a lagging number, but the trend across weeks leads the CPM schedule, because missed promises show up in the field long before they show up as recalculated dates.</p>



<p class="wp-block-paragraph">Store reasons for variance as categories, not free text. A PPC dip caused by weather tells you something different from three straight weeks of &#8220;material not on site.&#8221; The second one is a procurement problem that will keep producing delay until someone fixes it.</p>



<h3 class="wp-block-heading">Look-Ahead Churn</h3>



<p class="wp-block-paragraph">Count how many times an activity&#8217;s planned start has moved across snapshots of the three- or six-week look-ahead. An activity pushed out once is normal planning. One that has been pushed four updates running is being deferred, not planned, and the reason is usually a constraint nobody has named yet.</p>



<h3 class="wp-block-heading">Earned Value, Read Carefully</h3>



<p class="wp-block-paragraph">If you track earned value, the Schedule Performance Index (SPI = EV / PV) belongs on the dashboard, with a caveat many dashboards skip. Classic SPI drifts back toward 1.0 as the project nears completion, even when the job is late, because earned value and planned value both end at the budget at completion. A late-stage SPI that looks like recovery may just be arithmetic.</p>



<p class="wp-block-paragraph">Earned Schedule fixes this by measuring in time rather than money. It asks at what point in the baseline your current earned value was supposed to be reached, then divides that by the actual time elapsed to get SPI(t). It needs no new data collection, only different math on the same numbers. If you only plot one schedule index late in a job, plot SPI(t).</p>



<h3 class="wp-block-heading">Claimed Progress Versus Manpower</h3>



<p class="wp-block-paragraph">Daily logs record who was on site. When an activity&#8217;s reported percent complete keeps rising while the manpower logged against that trade stays well below the plan, one of those numbers is wrong. Usually it is the percent complete. This comparison is crude, and it needs a consistent trade mapping, but it is one of the few checks that uses data nobody is trying to make look good.</p>



<h2 class="wp-block-heading">Getting the Data Out of P6, Microsoft Project and Procore</h2>



<p class="wp-block-paragraph">A construction project dashboard lives or dies on boring extraction work. Plan for it.</p>



<h3 class="wp-block-heading">Primavera P6 XER Exports</h3>



<p class="wp-block-paragraph">An XER file is plain tab-delimited text. Lines beginning with <code>%T</code> name a table, <code>%F</code> lists its fields, and <code>%R</code> lines are rows. Activities live in the <code>TASK</code> table. This reader pulls one table into a list of dictionaries:</p>



<pre class="wp-block-code"><code>def read_xer_table(path, table_name):
    rows, fields, current = [], None, None
    # XER exports often use a Windows code page, not UTF-8
    with open(path, encoding="cp1252", errors="replace") as fh:
        for line in fh:
            parts = line.rstrip("rn").split("t")
            tag = parts[0]
            if tag == "%T":
                current = parts[1]
            elif tag == "%F" and current == table_name:
                fields = parts[1:]
            elif tag == "%R" and current == table_name:
                rows.append(dict(zip(fields, parts[1:])))
    return rows

tasks = read_xer_table("update.xer", "TASK")
for t in tasks[:5]:
    print(t["task_code"], t["task_name"], t["total_float_hr_cnt"])</code></pre>



<p class="wp-block-paragraph">Two details catch people. Float and durations in XER are stored in hours (<code>total_float_hr_cnt</code>), so converting to days means dividing by the hours per day of each activity&#8217;s calendar, not a blanket eight. And an activity with a value in <code>act_end_date</code> has an actual finish, which is a clean way to set <code>is_complete</code> without relying on status labels.</p>



<p class="wp-block-paragraph">Use the float P6 stored in the export rather than recomputing the critical path yourself. P6&#8217;s result reflects its scheduling options, constraints and calendars. Your recompute probably won&#8217;t.</p>



<h3 class="wp-block-heading">Microsoft Project</h3>



<p class="wp-block-paragraph">Microsoft Project can save a schedule as XML, which carries tasks, dates and total slack. The same snapshot logic applies. Save one file per status date, load it with its data date, and never replace the previous one.</p>



<h3 class="wp-block-heading">Procore and Other Project Management Platforms</h3>



<p class="wp-block-paragraph">Procore&#8217;s REST API exposes RFIs and submittals per project, including status, due dates and ball-in-court. Two warnings. Most useful endpoints are scoped to a single project, so a portfolio view means iterating over the project list. And submittal status values can be configured per company, so map them explicitly rather than hard-coding what you saw on one account. Autodesk Construction Cloud has its own RFI API with a similar shape.</p>



<h3 class="wp-block-heading">The Spreadsheet Nobody Wants to Admit Exists</h3>



<p class="wp-block-paragraph">The weekly work plan is very often an Excel file or a Smartsheet. Accept that. A scheduled job that reads a fixed-layout sheet from a shared folder is less elegant than an API and far more likely to get used. Fighting the field&#8217;s tools is how early warning systems die.</p>



<h2 class="wp-block-heading">Building the Dashboard and Alerts People Don&#8217;t Mute</h2>



<p class="wp-block-paragraph">For visualization I lean toward Grafana, self-hosted or on Grafana Cloud, because its alert rules can run SQL directly against PostgreSQL and route results to email, Slack or Microsoft Teams. You can see the style of dashboard in this <a href="https://monitor.john-nessime.com/public-dashboards/4c1463601230401ab55159059364b739" target="_blank" rel="noreferrer noopener">live Grafana demo</a>.</p>



<p class="wp-block-paragraph">Power BI deserves a fair hearing. Many contractors already license it, project controls teams know it, and its reporting and drill-down are excellent for monthly reviews. Where it is weaker is rule-based alerting on arbitrary queries. If your organization lives in Microsoft 365, a sensible split is Power BI for the reporting pack and a small scheduled job or Grafana instance for the alerts. Metabase is another reasonable middle ground for smaller teams.</p>



<p class="wp-block-paragraph">Whatever the tool, the alert design matters more than the charts:</p>



<ul class="wp-block-list">
<li><strong>Send a weekly digest, not a page.</strong> Schedule risk is not an outage. A Monday summary of every signal that crossed a threshold gets read. Twenty individual notifications get filtered.</li>



<li><strong>Name the owner in the alert.</strong> &#8220;RFI 142 blocks level 3 slab pour, need-by in 8 days, ball in court: structural engineer&#8221; is actionable. &#8220;Constraint risk: high&#8221; is not.</li>



<li><strong>Compare the project to itself.</strong> Portfolio-wide thresholds punish complex jobs and let simple ones coast. A drop from its own recent average is the better trigger.</li>



<li><strong>Alert on stale data.</strong> If no schedule update has loaded in longer than your cadence, that is itself a warning. An early warning dashboard showing old data is worse than none.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Float values look wrong compared to P6</h3>



<p class="wp-block-paragraph">Check the hours-to-days conversion first. Then check which float definition the schedule uses. A project-level &#8220;must finish by&#8221; date or activity constraints can produce negative float that looks like a bug but is correct. Also confirm whether the planner defines critical as longest path or as float below a threshold.</p>



<h3 class="wp-block-heading">Trend lines break after a re-baseline or renumbering</h3>



<p class="wp-block-paragraph">If activity IDs change, <code>LAG()</code> finds no previous row and the activity appears brand new. Add a check that counts activities present in the previous snapshot but missing from the current one. A sudden jump means the history needs a mapping table, not a fresh start.</p>



<h3 class="wp-block-heading">PPC sits at or near 100 percent every week</h3>



<p class="wp-block-paragraph">That usually means commitments are being made at the end of the week, or tasks are so loose they cannot fail. A PPC that never moves carries no signal. Look at how commitments are recorded before trusting the number.</p>



<h3 class="wp-block-heading">The dashboard is green but the field says you are behind</h3>



<p class="wp-block-paragraph">Check how percent complete is calculated. P6 supports duration, physical and units percent complete types, and they can tell very different stories about the same activity. Believe the field, then find which input is out of step.</p>



<h2 class="wp-block-heading">Common Mistakes</h2>



<ul class="wp-block-list">
<li>Building on the live schedule only, with no history, so every trend has to be reconstructed by hand.</li>



<li>Watching only the critical path and missing the near-critical work that becomes critical next month.</li>



<li>Trusting classic SPI in the final third of a project, when it drifts toward 1.0 regardless of performance.</li>



<li>Tracking overdue RFIs without linking them to the activities they block.</li>



<li>Recomputing float yourself instead of using what the scheduling tool calculated.</li>



<li>Sending so many alerts that the project team creates an inbox rule to hide them.</li>
</ul>



<h2 class="wp-block-heading">Best Practices</h2>



<ul class="wp-block-list">
<li>Append every export with its data date. Never overwrite history.</li>



<li>Start with two signals, float erosion and constraint aging, and add more once those are trusted.</li>



<li>Agree with the planner on update cadence. The dashboard cannot be fresher than its inputs.</li>



<li>Keep the RFI-to-activity mapping small and owned by one named person.</li>



<li>Review thresholds after the first month. Early numbers are guesses.</li>



<li>Show each signal with its source and data date, so nobody argues about whether it is current.</li>
</ul>



<h2 class="wp-block-heading">Frequently Asked Questions</h2>



<h3 class="wp-block-heading">What are leading indicators of construction delay?</h3>



<p class="wp-block-paragraph">The most useful are float erosion on near-critical activities, RFIs and submittals aging toward the start of the work they block, a falling PPC trend, activities repeatedly pushed out of the look-ahead, and reported progress outpacing logged manpower.</p>



<h3 class="wp-block-heading">How often should the schedule be updated for early warning to work?</h3>



<p class="wp-block-paragraph">More often than monthly if you can manage it. Weekly or biweekly updates give the trend queries enough points to show movement. Signals from RFIs and weekly plans can refresh daily regardless of schedule cadence.</p>



<h3 class="wp-block-heading">Is SPI a good early warning metric?</h3>



<p class="wp-block-paragraph">Early and mid-project, yes, alongside float and critical path analysis. Late in a project, classic SPI drifts toward 1.0 even on late jobs, so the time-based SPI(t) from Earned Schedule is more trustworthy.</p>



<h3 class="wp-block-heading">Can I build this with Power BI instead of Grafana?</h3>



<p class="wp-block-paragraph">Yes for the reporting side, and it is often the better choice where the team already uses it. The snapshot database and trend logic are the same either way. Rule-based alerting is where you may want a separate tool.</p>



<h3 class="wp-block-heading">Do I need Primavera P6 for a construction early warning dashboard?</h3>



<p class="wp-block-paragraph">No. Any scheduling tool that exports activities with dates and float works, including Microsoft Project. What you need is a consistent export on every update and somewhere to keep the history.</p>



<h3 class="wp-block-heading">How do I link RFIs to schedule activities?</h3>



<p class="wp-block-paragraph">Usually with a small mapping table maintained by the project engineer: RFI or submittal number, activity ID, and need-by date. Mapping by area and spec section is a workable first step if activity-level links are too much at the start.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">The project manager who learns about delay three weeks late usually isn&#8217;t short of data. The problem is that the data is spread across a schedule, an RFI log and a weekly plan, and each one only remembers its latest state. A construction early warning dashboard works because it keeps the history and watches the movement: float draining from work that isn&#8217;t critical yet, answers arriving after the work that needs them, promises slipping week after week. Snapshot first. Everything else is a query on top.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need Help Building a Construction Early Warning Dashboard?</h2>



<p class="wp-block-paragraph">I build and host the data side of this for construction teams and project controls groups. Here is where I can help:</p>



<ul class="wp-block-list">
<li>Setting up a snapshot database that stores every P6 or Microsoft Project update with its data date.</li>



<li>Writing XER and XML loaders that handle hours-based float, calendars and renumbered activities.</li>



<li>Pulling RFIs and submittals from Procore or Autodesk Construction Cloud and linking them to activities.</li>



<li>Building Grafana dashboards for float erosion, constraint aging, PPC and SPI(t) trends.</li>



<li>Designing weekly digest alerts with named owners, sent to email, Slack or Teams.</li>



<li>Hosting and maintaining the dashboard so your team reads it instead of running it.</li>
</ul>



<p class="wp-block-paragraph">If you want to scope it first, the <a href="https://john-nessime.com/tools/engagement-builder/">Engagement Builder</a> lays out phases and effort. Or send me one recent XER export and your RFI log, and I will tell you which signals your data can already support.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/hosted-monitoring/">See Hosted Monitoring</a></div>



<div class="wp-block-button is-style-outline is-style-outline--1"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/construction-early-warning-dashboard/">The Delay Was in the Data Three Weeks Ago: Building a Construction Early Warning Dashboard</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Designing a Construction Dispute Intelligence Platform That Survives Cross-Examination</title>
		<link>https://john-nessime.com/blog/technical-guides/construction-dispute-intelligence-platform/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 20:19:52 +0000</pubDate>
				<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[As-Built Schedule]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[Bates Numbering]]></category>
		<category><![CDATA[Capital Projects]]></category>
		<category><![CDATA[Chain of Custody]]></category>
		<category><![CDATA[Common Data Environment]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Delay Analysis]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Embeddings]]></category>
		<category><![CDATA[Extension of Time]]></category>
		<category><![CDATA[Forensic Schedule Analysis]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Hybrid Search]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[Load Files]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Oracle Aconex]]></category>
		<category><![CDATA[pgvector]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[Project Controls]]></category>
		<category><![CDATA[Provenance]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Reciprocal Rank Fusion]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[System of Record]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<category><![CDATA[XER Files]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=336</guid>

					<description><![CDATA[<p>A layer-by-layer architecture for a construction dispute intelligence platform: hashed intake and immutable originals, OCR with page coordinates, a versioned Primavera P6 schedule plane, an event model that joins documents to activities, hybrid retrieval on pgvector, and a generation layer bounded by human confirmation. Written around what each layer prevents.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/construction-dispute-intelligence-platform/">Designing a Construction Dispute Intelligence Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that kills these systems is always the same one, and it always comes from the expert, not the engineer: &#8220;Where did this date come from?&#8221;</p>



<p class="wp-block-paragraph">You have a screen showing a beautiful chronology. Two hundred events, colour-coded by discipline, cross-referenced against the programme. The delay expert points at row 47, the one that says the piling rig arrived late, and asks which document that came from, who produced it, what page, and whether the other side has seen it. If the answer involves anyone opening a folder and searching, the platform has already failed. Not technically. Evidentially. A chronology nobody can trace back to a produced document is a nice picture, and a nice picture is worth nothing in an adjudication.</p>



<p class="wp-block-paragraph">That is the constraint that shapes everything else. A construction dispute intelligence platform is not a search engine with a construction theme on it. It is a provenance system that happens to do search. Get that ordering wrong and you build something impressive that no expert will sign their name to.</p>



<p class="wp-block-paragraph">This post walks through the architecture layer by layer: intake and chain of custody, normalization, the schedule data plane, the chronology model that joins the two, retrieval, and the tightly bounded generation layer on top. For each one I&#8217;ll say what it prevents, because that is the only reason any of these layers exist.</p>



<h2 class="wp-block-heading">What a construction dispute intelligence platform actually has to do</h2>



<p class="wp-block-paragraph">Strip away the vocabulary and the job is narrow. Take a pile of heterogeneous records from a capital project, some of them structured and some of them scanned faxes, and let a small number of expensive humans answer three questions quickly and defensibly:</p>



<ul class="wp-block-list">
<li>What happened, in what order, and who said so at the time?</li>



<li>What did the programme say before and after each of those things happened?</li>



<li>Which contemporaneous record supports every single assertion in the report?</li>
</ul>



<p class="wp-block-paragraph">The third one is not a nice-to-have. Both of the industry references people actually argue from lean on it. The Society of Construction Law Delay and Disruption Protocol raised record keeping to a core principle in its second edition and stopped expressing a preference between retrospective delay analysis methods, on the reasoning that the records you have largely determine the method available to you. AACE International&#8217;s Recommended Practice 29R-03 codifies forensic schedule analysis into a taxonomy of Method Implementation Protocols, and its minimum implementation protocols are essentially statements about which source schedules you must be able to produce. Both documents are, underneath, about evidence handling. So is your architecture.</p>



<h2 class="wp-block-heading">Layer one: intake that a forensic expert can defend</h2>



<p class="wp-block-paragraph">Everything upstream of this layer is somebody else&#8217;s mess. Site photos on a phone, a shared drive nobody pruned since mobilization, an Aconex export, a hard drive that arrived in an envelope. Your job at intake is to make an irreversible statement about what you received and when, before anyone touches it.</p>



<p class="wp-block-paragraph">Hash every file on arrival and store the hash next to the object, not in a spreadsheet. SHA-256 over the raw bytes, computed before any conversion, deduplication or renaming. The hash becomes the document&#8217;s identity for the rest of its life.</p>



<pre class="wp-block-code"><code># Build an arrival manifest before anything else runs
find ./ACME-INTAKE-001 -type f -print0 
  | xargs -0 sha256sum &gt; manifest-ACME-INTAKE-001.txt

# Re-verify later, after the files have moved through processing
sha256sum -c manifest-ACME-INTAKE-001.txt</code></pre>



<p class="wp-block-paragraph">Then put the originals somewhere they cannot be quietly edited. On AWS, S3 Object Lock in compliance mode with a retention period is the blunt instrument: once set, the object version cannot be overwritten or deleted by anyone, including the root account, until retention expires. Governance mode is the softer variant, where a specifically permissioned principal can lift the lock, which is fine for internal working copies and wrong for originals. If you are self-hosting on a Contabo or InterServer box because the client will not put project data in a hyperscaler, the equivalent discipline is a write-once staging volume plus an append-only manifest log, and you accept that you are relying on process rather than on the storage layer refusing the write.</p>



<h3 class="wp-block-heading">The manifest is the product</h3>



<p class="wp-block-paragraph">If the matter is already in formal disclosure, you will be handed productions rather than raw folders, and those arrive as load files. A Concordance DAT carries one row per document with fields like BegBates, EndBates, Custodian, BegAttach, EndAttach and an MD5 hash. An Opticon OPT carries one row per page, mapping Bates numbers to image paths. The delimiters are deliberately weird characters that real text rarely contains, and different tools disagree about the exact code points, so parse defensively and log what you detected rather than assuming.</p>



<p class="wp-block-paragraph">The failure mode here is quiet and expensive. Variable-width Bates numbers sort lexically instead of numerically, so ABC10 lands between ABC1 and ABC2, attachment ranges resolve to the wrong parents, and a family of documents silently splits. Nobody notices until an expert cites a covering email whose attachment, in your system, belongs to a different letter. Pad to fixed width on ingest and reconcile every BegAttach range against the documents you actually loaded.</p>



<h2 class="wp-block-heading">Layer two: normalization, and the OCR tax nobody budgets for</h2>



<p class="wp-block-paragraph">Construction corpora are the worst kind of mixed. Native Office files with clean text, born-digital PDFs, scanned site instructions with a signature and a coffee ring, photographs of whiteboards, and drawings where the only text that matters is inside a title block in the bottom right corner.</p>



<p class="wp-block-paragraph">Route by what the file actually is, not by extension. For a PDF, try text extraction first; if the extracted layer is empty or nearly empty, it is a scan and needs OCR. Tesseract via OCRmyPDF is the cheap path and it is genuinely good on clean typed pages. It degrades badly on handwriting, on stamps, and on anything skewed, which describes most site paperwork. A managed service such as Amazon Textract handles tables and forms better and returns geometry for every detected block, which matters more than accuracy alone.</p>



<p class="wp-block-paragraph">Geometry is the part teams skip and then rebuild six months later. If you store only the extracted string, you can prove a phrase exists somewhere in a document. If you store the page number and the bounding box, you can put a highlight on the exact line in the viewer, and the expert can confirm it in two seconds instead of reading eleven pages. That difference is the difference between a tool people use and a tool people abandon.</p>



<p class="wp-block-paragraph">Keep the OCR text separate from the original in storage, and record which engine and which settings produced it. When a disputed reading of a handwritten date turns up, and it will, you want to be able to say exactly how that character was produced.</p>



<h2 class="wp-block-heading">Layer three: the schedule plane is a separate data model</h2>



<p class="wp-block-paragraph">This is where most teams underestimate the work. Programme data is not documents. It is a graph, versioned, with a calendar attached, and it needs its own tables, its own ingest path and its own validation.</p>



<p class="wp-block-paragraph">Primavera P6 exports as XER, a tab-delimited text format with a table-per-section layout, or as P6 XML. Either way you are pulling out projects, the WBS tree, activities, logic links and calendars. In XER these live in sections including PROJECT, PROJWBS, TASK, TASKPRED and CALENDAR. Parsing it by hand is a bad use of a week; PyP6Xer is an open source Python parser that will get you to objects quickly.</p>



<pre class="wp-block-code"><code>from xerparser.reader import Reader

xer = Reader("update-14.xer")

for project in xer.projects:
    print(project.proj_short_name)
    for activity in project.activities:
        print(activity.task_code, activity.task_name)</code></pre>



<p class="wp-block-paragraph">Two things bite immediately. First, several duration and float fields in XER are stored in hours rather than days, which you can see in the column names themselves. If you load them straight into a &#8220;days&#8221; column because the numbers looked plausible on a five-day calendar, every float figure downstream is wrong by a factor that varies with the calendar. Check your own export before you trust any mapping, including this one.</p>



<p class="wp-block-paragraph">Second, activity IDs are not stable across a project&#8217;s life. Activities get renumbered, split, deleted and re-added between updates. If your schema keys on the activity ID alone, two updates will silently merge activities that were never the same work. Key on a composite of schedule version plus activity ID, and maintain a separate mapping table for identity across versions that a human can correct. That table is boring, it is not automatable, and it is the difference between a windows analysis that holds together and one that quietly compares apples to a different orchard.</p>



<p class="wp-block-paragraph">Store every update as an immutable snapshot with its data date. Never mutate a loaded schedule. The whole point of the layer is that you can reconstruct exactly what the programme said on a given day, which is precisely what a time slice or windows method needs.</p>



<h2 class="wp-block-heading">Layer four: the chronology is the join, and the join is the hard part</h2>



<p class="wp-block-paragraph">Documents on one side, schedule snapshots on the other. The product is the thing in the middle: an event, with a date, a description, a set of supporting document references, and a set of affected activities.</p>



<p class="wp-block-paragraph">Model it explicitly. An event is not a document and it is not an activity. One letter can create three events. One event can be evidenced by nine documents and a photograph. The relationship is many-to-many in both directions and every attempt I have seen to flatten it ends with someone maintaining a spreadsheet on the side.</p>



<pre class="wp-block-code"><code>-- Every assertion resolves to a page and a box on that page
CREATE TABLE evidence_ref (
  id            BIGSERIAL PRIMARY KEY,
  event_id      BIGINT NOT NULL REFERENCES event(id),
  document_id   UUID   NOT NULL REFERENCES document(id),
  page_number   INT    NOT NULL,
  bbox          JSONB,             -- normalized coordinates from OCR
  quoted_text   TEXT   NOT NULL,
  asserted_by   TEXT   NOT NULL,   -- 'analyst' or 'model'
  confirmed_by  TEXT,              -- NULL until a human confirms
  confirmed_at  TIMESTAMPTZ
);</code></pre>



<p class="wp-block-paragraph">Those last three columns are the ones that matter. They encode the difference between something the machine proposed and something a person stands behind. Report generation reads only rows where <code>confirmed_by</code> is set. Everything else is a suggestion queue. This is not a compliance box; it is the mechanism by which an expert can put their name on the output, and it needs to exist in the schema rather than in a workflow convention that erodes under deadline.</p>



<h2 class="wp-block-heading">Layer five: retrieval that finds document numbers, not just meaning</h2>



<p class="wp-block-paragraph">Pure vector search is the wrong default for this corpus, and it fails in a way that looks like success. Embeddings are good at meaning and bad at tokens. Ask for &#8220;RFI 0442&#8221; and a dense retriever will cheerfully hand you eleven chunks about requests for information, none of which is RFI 0442, because the identifier got smeared into its semantic neighbourhood. Construction disputes run on identifiers: RFI numbers, variation order references, drawing revisions, activity IDs, Bates ranges.</p>



<p class="wp-block-paragraph">Run lexical and semantic retrieval in parallel and fuse the rankings. PostgreSQL with the pgvector extension does both without adding infrastructure: an HNSW index over the embedding column, a GIN index over a generated tsvector column, and reciprocal rank fusion to merge the two candidate lists.</p>



<pre class="wp-block-code"><code>CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunk (
  id          BIGSERIAL PRIMARY KEY,
  document_id UUID NOT NULL,
  page_number INT  NOT NULL,
  body        TEXT NOT NULL,
  body_tsv    tsvector GENERATED ALWAYS AS
                (to_tsvector('english', body)) STORED,
  embedding   vector(1536)
);

CREATE INDEX chunk_tsv_idx  ON chunk USING gin (body_tsv);
CREATE INDEX chunk_hnsw_idx ON chunk USING hnsw (embedding vector_cosine_ops);</code></pre>



<p class="wp-block-paragraph">Over-fetch from each side and trim after fusion rather than pulling the final count from each; more candidates give the fusion step something to work with. Add a trigram index if your corpus has OCR noise, because &#8220;RFI 0442&#8221; and &#8220;RFl 0442&#8221; are the same document to a human and different tokens to Postgres.</p>



<p class="wp-block-paragraph">Filter hard on metadata before ranking. Matter, custodian, date range, privilege status. A retrieval layer that can return a privileged document into a shared workspace is a liability, not a feature.</p>



<h2 class="wp-block-heading">Layer six: generation, bounded hard</h2>



<p class="wp-block-paragraph">Language models are useful here and the useful jobs are smaller than the demos suggest: proposing candidate events from a batch of correspondence, drafting neutral one-line summaries, clustering near-duplicate records, suggesting which activities an event might touch. Every one of those is a proposal that lands in a review queue.</p>



<p class="wp-block-paragraph">The reason to be strict is not squeamishness. Stanford&#8217;s RegLab and HAI ran a preregistered evaluation of commercial legal research tools built on retrieval-augmented generation, with expert-scored answers, and found hallucination rates in the range of roughly 17 to 33 percent depending on the product, against vendor marketing that had claimed hallucination-free citations. The vendors disputed the characterization. The finding that should worry you more than the headline number is the failure mode underneath it: a citation that exists, is quoted accurately, and does not stand for what the answer says it stands for. Retrieval reduces fabrication. It does not produce correct reasoning about what a document means, and misgrounding is far harder to spot than an invented reference.</p>



<p class="wp-block-paragraph">So bound it. Generation writes to the proposal table, never to the report. Every generated assertion carries the chunk IDs it was built from, and the UI shows the source page beside the claim by default rather than behind a click. If a proposed event has no evidence reference, it does not enter the queue at all.</p>



<h2 class="wp-block-heading">Tenancy, privilege, and the thing that ends the engagement</h2>



<p class="wp-block-paragraph">Matters do not share data. Ever. Not &#8220;logically separated by a WHERE clause that we always remember to add&#8221;. The cheapest defensible pattern is a database per matter, or at minimum a schema per matter with connection-level scoping, so that a missing predicate produces an empty result rather than another client&#8217;s correspondence.</p>



<p class="wp-block-paragraph">Two more that get discovered late. Privilege state is per document per matter and it changes; model it as a versioned attribute with an audit trail, not a boolean someone flipped. And plan the exit at the start: matters end, retention clocks expire, and someone has to certify destruction. That means knowing every copy you made, including OCR text, thumbnails, embeddings and backups. Tools like the O&amp;O Software utilities have a place at the endpoint end of that job; the platform side is a deletion manifest that enumerates derived artifacts per document, written at ingest rather than reconstructed under pressure.</p>



<p class="wp-block-paragraph">If experts are working from home, from site cabins, or from three countries, the access path is part of the design. A managed VPN such as NordVPN or Surfshark on the client side plus network-level allowlisting on the platform side is a pragmatic combination, and it is worth writing into the engagement documentation so nobody improvises with a personal Dropbox at eleven at night.</p>



<h2 class="wp-block-heading">Where these platforms break in practice</h2>



<ul class="wp-block-list">
<li><strong>Chronology dates drift from document dates.</strong> Usually a timezone or a date-format collision: a scanned form read as DD/MM, an email header parsed as UTC, a schedule data date stored as a naive timestamp. Store the raw string you extracted alongside the parsed value and show both in the review UI.</li>



<li><strong>Float figures look wrong by a constant factor.</strong> Hours versus days in the schedule import. Check against the same activity opened in P6 before you debug anything else.</li>



<li><strong>Retrieval misses a document everyone knows exists.</strong> Either it was never OCR&#8217;d because the PDF had a thin text layer that passed your emptiness check, or the identifier is only in the title block of a drawing. Log extraction character counts per page and alert on suspiciously low ones.</li>



<li><strong>Two activities merged across updates.</strong> Composite keying was not applied, or an activity was renumbered. Diff activity code sets between consecutive snapshots and surface additions and deletions as a report the planner reviews.</li>



<li><strong>Document families split.</strong> Bates padding or an unreconciled attachment range from the load file. Re-run the family reconciliation and count orphaned attachments.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the schedule as documents. It is a graph with versions, and shoehorning it into the document store costs you every analysis you actually wanted.</li>



<li>Storing extracted text without page and coordinate provenance. Cheap now, a full reprocessing run later.</li>



<li>Vector-only retrieval. Looks great in a demo built from prose, falls over on the first RFI number.</li>



<li>Letting model output reach a deliverable without a recorded human confirmation.</li>



<li>Mutating loaded schedules to &#8220;fix&#8221; them. Fix on the way in, or hold the correction as an annotation. Never edit the snapshot.</li>



<li>Deferring the deletion story to the end of the matter, when the derived artifacts are already scattered.</li>
</ul>



<h2 class="wp-block-heading">Best practices for building a construction dispute intelligence platform</h2>



<ul class="wp-block-list">
<li>Hash at the door, lock the originals, and never process in place.</li>



<li>Make every derived artifact point back to a document ID, a page and a region.</li>



<li>Give schedules their own immutable, version-keyed store with the data date as a first-class field.</li>



<li>Model events separately from both documents and activities, with many-to-many links in each direction.</li>



<li>Run hybrid retrieval and fuse the rankings; filter on matter and privilege before you rank.</li>



<li>Keep generation in a proposal queue with confirmation recorded in the schema.</li>



<li>Isolate matters at the database or schema boundary, not in application code.</li>



<li>Instrument ingest as seriously as you would a production pipeline. Grafana over your extraction counts and OCR queue depth will tell you a corpus is going wrong days before an analyst does.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a dedicated vector database for this?</h3>



<p class="wp-block-paragraph">Almost certainly not. Matter corpora are large in gigabytes and small in chunk count compared to consumer-scale search. PostgreSQL with pgvector keeps the embeddings in the same transaction as the provenance rows, which means you cannot end up with a chunk whose document record was rolled back. That consistency is worth more here than the last few milliseconds of query latency.</p>



<h3 class="wp-block-heading">Should the platform pick a delay analysis method?</h3>



<p class="wp-block-paragraph">No. That choice belongs to the expert and depends on the contract, the forum, the records available and the timing of the analysis. What the platform should do is make it obvious which methods the record set can support, by reporting what you actually have: how many schedule updates, at what intervals, with what data dates, and how complete the contemporaneous correspondence is around each window.</p>



<h3 class="wp-block-heading">XER or P6 XML for schedule ingest?</h3>



<p class="wp-block-paragraph">Accept both, because you will be sent both. XER is what planners export by habit and it is easy to parse with existing libraries. P6 XML is more explicit about structure and less prone to encoding surprises. Whichever arrives, keep the original file, hashed, alongside the parsed rows.</p>



<h3 class="wp-block-heading">How much of the review can realistically be automated?</h3>



<p class="wp-block-paragraph">The ranking and the grouping, most of it. The judgement, none of it. Automation that reduces a hundred thousand documents to a few thousand worth reading is enormously valuable and uncontroversial. Automation that decides what a letter means is where you inherit risk you cannot price.</p>



<h3 class="wp-block-heading">Can I build this on top of a common data environment we already have?</h3>



<p class="wp-block-paragraph">Partly. A CDE such as Aconex is a good source of records and a poor system of record for a dispute, because access changes, exports are point-in-time, and the retention rules belong to the project rather than to the matter. Pull from it, hash what you pulled, and hold your own copy under your own lock.</p>



<h3 class="wp-block-heading">What does this cost to run?</h3>



<p class="wp-block-paragraph">The shape of the bill is more useful than a number. OCR and embedding are one-time per document and scale with page count, so a scan-heavy corpus costs multiples of a born-digital one. Storage is small and constant. Retrieval is negligible. Generation scales with how much you push through it, which is another argument for narrow, queued use rather than a chat box over the whole corpus.</p>



<h3 class="wp-block-heading">Is a general eDiscovery platform not enough?</h3>



<p class="wp-block-paragraph">For document review, often yes, and they are considerably better at productions and privilege logging than anything you will build in a quarter. What they do not have is the schedule plane. The value of a construction-specific build is the join between correspondence and programme versions, so a sensible architecture frequently uses an established review platform for documents and adds the schedule and chronology layers alongside it.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A construction dispute intelligence platform earns its place by making provenance cheap. Search, clustering and summarization are commodity capabilities now; anyone can bolt them onto a corpus in a fortnight. What is hard, and what decides whether an expert will rely on the output, is that every date, every event and every sentence in the eventual report resolves in one click to a specific page of a specific produced document, with a record of who confirmed it.</p>



<p class="wp-block-paragraph">Build the provenance spine first. Hash at intake, keep coordinates, version the schedules, model events as their own thing, and keep the model on the proposal side of a human confirmation. The clever features are easy to add afterwards. Retrofitting evidence handling into a system that was designed as a search tool is a rebuild, and it always arrives at the worst possible moment, which is about three weeks before a hearing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or fixing this?</h2>



<p class="wp-block-paragraph">I work on the infrastructure side of exactly this problem: getting messy project records into a system where every claim is traceable. Things I can help with:</p>



<ul class="wp-block-list">
<li>Designing the intake and chain-of-custody layer, including hashing, immutable storage, and load file parsing that survives odd delimiters and inconsistent Bates padding.</li>



<li>Building the schedule ingest path for XER and P6 XML, with version-keyed snapshots and an activity identity map that holds across updates.</li>



<li>Standing up hybrid retrieval on PostgreSQL and pgvector, including page-level and coordinate-level provenance so highlights land on the right line.</li>



<li>Setting up the OCR and normalization pipeline, with routing rules, quality checks, and alerting on pages that extracted suspiciously little text.</li>



<li>Matter isolation, privilege state modelling, audit logging, and a deletion manifest you can act on at the end of the engagement.</li>



<li>Reviewing an existing platform and telling you plainly where the provenance chain breaks.</li>
</ul>



<p class="wp-block-paragraph">If you have something concrete, send it over: an XER export, a sample DAT and OPT pair, a page count breakdown of your corpus, or the schema you are currently working from. It is much easier to say something useful about a real file than about a description of one.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/construction-dispute-intelligence-platform/">Designing a Construction Dispute Intelligence Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
