Two source records had been collapsed into one. Flat 12 and Flat 12A in the same block: same postcode, same rooftop geocode, both two-bed, both published the same week by different agencies. One record won the price field. The other won the contact block. What came out the far end was a listing for a property that does not exist.

That is what makes property listing deduplication hard, and it is not the problem people plan for. This post covers candidate generation, match scoring, survivorship rules, the time axis almost nobody models, and how each fails in production.

Under-merging is visible. Over-merging is not.

Get this asymmetry wrong and every threshold downstream is mistuned.

Fail to merge two records for the same property and a user sees two cards for one house. It looks sloppy, someone reports it, you fix it that afternoon.

Merge two different properties and nothing looks broken. The card renders. Real price, real photos, real agent, all belonging to different homes. Filters return it for the wrong bedroom count. Price analytics inherit a value never asked for that unit. A lead goes to an agency that never had the instruction. Nobody reports it, because nobody can see it.

A missed merge is a duplicate. A false merge is corrupted data with a plausible face on it.

Why property listing deduplication is not a string-matching problem

The obvious first attempt is to normalise the address, hash it, and group by the hash. It gets you further than you expect and then stops dead, because portals do not disagree on addresses randomly. They disagree structurally.

• One portal puts the sub-unit in its own field, one puts it in line one, one drops it.
• New-build stock is marketed by plot number long before it is marketed by street address.
• Rural properties are named rather than numbered, and the name is what agencies edit most.
• Some agencies fuzz the address on tenanted stock and publish only a street or postcode sector.

Normalisation still earns its place, it just is not the decision. libpostal is the standard tool: a C library trained on open address data, with bindings for Python, Go, Ruby, Java and Node, plus a PostgreSQL extension.

-- Via the pgsql-postal extension, which wraps libpostal.
-- postal_normalize returns every expansion that "makes sense",
-- so you index all of them rather than betting on one.
SELECT unnest(postal_normalize('Flat 12a, 12 Bramble Ct., Manchester'));

Expand rather than collapse. You do not know which spelling the other portal chose, and matching on set intersection is more forgiving than picking a canonical form and hoping both sides picked the same one.

Look for a real identifier first

Probabilistic matching is what you do when there is no key. Sometimes there is one, and it is worth a lot of scoring code.

• Great Britain: the UPRN. Assigned to every addressable location by local authorities and Ordnance Survey, it survives renaming, renumbering and demolition. OS Open UPRN publishes identifiers and coordinates as open data. Same UPRN on both sides ends the argument.
• United States: inside a single MLS, the RESO Data Dictionary gives a stable listing key and the RESO Web API is the current transport. RETS is deprecated and boards have been switching it off. The key is stable within an MLS, not across them, so overlapping markets put you back into probabilistic matching.
• Open cadastre: a parcel or title identifier is the best anchor available, with the caveat that one parcel can hold many dwellings.

Resolve on the identifier first and let the fuzzy matcher handle the remainder. That set is smaller and stranger than you expect, which is fine, because the strange ones are where false merges live.

Blocking: never compare every record with every record

All-pairs comparison is quadratic. A million listings is roughly half a trillion pairs. You do not need a bigger machine, you need to stop generating pairs that were never going to match. Blocking produces candidates cheaply from an index, and only candidates get scored.

A good key is selective enough to shrink the space and stable enough that both portals agree on it. Those pull against each other. Postcode plus bedroom count is selective, but bedroom count is one of the most disputed fields in any feed, because portals count studies, box rooms and loft conversions differently. Block on it and you discard true matches before scoring runs.

Keys that hold up tend to be geometric or structural rather than descriptive:

• A geohash or H3 cell around the geocoded point, plus its neighbours, so pairs straddling a cell edge survive.
• Postcode or postal sector combined with the numeric part of the street address.
• A trigram index on the normalised street name, which tolerates abbreviations that break exact keys.
• The agency contact number, which catches one agency’s stock syndicated under wildly different formatting.
• A perceptual hash of the lead photo, which catches what the address fields lost.

CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;

CREATE INDEX ON listing USING gist (geog);
CREATE INDEX ON listing USING gin (street_norm gin_trgm_ops);

-- The % operator compares against pg_trgm.similarity_threshold,
-- which is a session setting, not a hardcoded constant.
SET pg_trgm.similarity_threshold = 0.6;

-- Candidate pairs only. b.id > a.id yields each pair once.
SELECT a.id AS left_id, b.id AS right_id
FROM listing a
JOIN listing b
  ON b.id > a.id
 AND ST_DWithin(a.geog, b.geog, 75)
 AND a.street_norm % b.street_norm
WHERE a.source_portal <> b.source_portal;

ST_DWithin on a geography column takes metres and uses the GiST index. Seventy-five is deliberately generous: rooftop and street-interpolated geocodes for one address routinely land tens of metres apart. Strictness belongs in the score, not the block.

Every key has a blind spot, so run three or four narrow passes and union the results. That is far cheaper than one pass loose enough to cover everything. Track what each pass contributes alone, because passes rot: a feed changes, a pass stops producing candidates, and recall drops without a single error in the logs.

Scoring: what each field is actually worth

The instinct is to weight fields by how much you trust them. That is the wrong axis. The right one is how surprising the agreement is.

This is the Fellegi-Sunter model, the standard framing for record linkage. Two probabilities matter per comparison: how likely the fields agree given the records are the same property, and how likely they agree given they are not. The ratio is the evidence. Multiply across fields for a match score.

Cardinality decides the weight

Two listings agreeing on “detached house” tells you almost nothing, because a large share of the market is detached houses. Agreeing on an unusual property name in a small postcode sector tells you a great deal, because there are very few ways for that to happen by accident.

So a field’s weight is not constant. Agreeing on a postcode in a dense city centre is weak evidence; the same agreement in a sparse rural area is strong. Term-frequency adjustment handles that per value rather than per column, which matters in property data because property data is geographically lumpy.

Splink is the one I reach for first. It implements Fellegi-Sunter with expectation-maximisation parameter estimation, runs on DuckDB locally and Spark for larger jobs, supports term-frequency adjustments, and shows which field contributed what to a score. That last part matters: “the model said so” is not an answer you can act on. The trade-off is that an unsupervised model needs no labelled data but is only as sane as your blocking.

Photos are strong evidence until they aren’t

Two portals carrying one property usually carry the same photo set from the same agency, occasionally recompressed. A perceptual hash survives that, because it fingerprints visual structure rather than bytes.

from PIL import Image
import imagehash

# hash_size=16 gives a 256-bit hash instead of the 64-bit default.
# More bits means fewer accidental collisions between two beige
# living rooms shot from the same corner with the same wide lens.
a = imagehash.phash(Image.open("portal_a_01.jpg"), hash_size=16)
b = imagehash.phash(Image.open("portal_b_03.jpg"), hash_size=16)

distance = a - b   # subtraction returns the Hamming distance
print(distance)

Use the count of near-identical pairs across both sets, not the best single pair. One shared photo can be a stock exterior. Six is not.

Now the trap that produces false merges at scale. New-build schemes reuse the same CGI renders, show-home interiors and drone shots across every plot. Photo evidence will insist that forty distinct apartments are one apartment, and it will insist confidently, because the agreement really is surprising.

Treat photo agreement as conditional. Count how many distinct listings share each hash across the corpus, then downweight or discard hashes appearing on more than a handful. Same term-frequency logic, applied to images. Watermarks are the other tripwire: a portal stamping its logo into the corner shifts the hash for every photo it serves.

Three outcomes, not two

A score comes out. What you do with it should not be a single threshold.

1. Below the lower threshold: treat as distinct. Two cards.
2. Between the thresholds: hold for review. Do not merge, do not discard, surface the pair with its field-level evidence.
3. Above the upper threshold: merge automatically.

The width of that middle band is a budget decision, not an accuracy decision. Widen it and you catch more ambiguous pairs before they do damage, at the cost of review time. Narrow it and you are choosing which error you will eat. Because over-merging is the expensive one, the band should sit asymmetrically.

One rule with no downside: clustered records keep their source rows. Store the cluster assignment in a separate table keyed by source record ID, never by overwriting the source. When a merge turns out wrong, you want an UPDATE on a mapping table rather than a restore from backup.

Survivorship: which record wins each field

Merging decides records belong together. Survivorship decides what the merged record says, and it is where the opening bug came from. Do not pick a winning record. Pick a winning source per field, and write it down.

• Price: most recently observed, tie-broken by source rank. Staleness is the main risk.
• Agent and contact: highest-ranked source only, never blended. A number and an agency name are one unit; splitting them produces a contact that does not exist.
• Floor area and bedrooms: the source with the most complete structured fields, and flag the cluster when sources disagree. Disagreement here often means the merge itself is wrong.
• Photos: union, deduplicated by hash, ordered by highest-ranked source.

-- Winning price per cluster: freshest observation,
-- with source_rank breaking ties on identical timestamps.
SELECT DISTINCT ON (cluster_id)
       cluster_id,
       price,
       source_portal,
       observed_at
FROM listing_price_observation
ORDER BY cluster_id,
         observed_at DESC,
         source_rank ASC;

Keep provenance on the output. Every surviving field should carry which source it came from and when it was observed. That is the difference between debugging a bad merge in ten minutes and debugging it in two days.

The time axis nobody models

A property is listed, withdrawn, then relisted months later with a different agency, a different price and new photos. One entity or two?

Both. There are two entities and most pipelines model only one:

• The property: physical and permanent. Same building, same UPRN or parcel, forever.
• The listing event: commercial and episodic. One campaign, one agency, one price band.

Collapse them and you get a record whose price history is a jagged line with no explanation, whose days-on-market figure is meaningless, and where a price cut and a relisting are indistinguishable. Separate them and both questions get easy: duplicates resolve at the property level, campaign analytics at the event level.

My rule for splitting events: a gap in observation beyond some window, combined with a change of agency or a price move outside a tolerance, starts a new event on the same property. Tune the window per market.

Troubleshooting: where these pipelines break

Cluster count collapses overnight

Almost always a blocking key that became non-selective. A portal starts sending a placeholder coordinate for missing geocodes, ten thousand listings land on one point, and that becomes a single enormous block. Alert on largest-block size, not just job duration. A pass that generates one huge block runs slowly first and merges wrongly second.

One cluster swallows a whole street

Transitive closure doing what it was told. A matches B, B matches C, so all three are one cluster even though A and C are plainly different. Cap cluster size and quarantine anything over it. Check internal consistency too: a cluster with three bedroom counts and a fifty percent price spread is wrong regardless of the pairwise scores.

Fresh duplicates appear every single day

The source record IDs are not stable. A portal that regenerates its internal ID on every republish produces a new record daily, and a cluster mapping keyed on that ID re-resolves from scratch every run. Key the mapping on a fingerprint you control, built from fields that do not churn.

The run got slow and nothing changed

Check the trigram and GiST indexes are still being used. A change in normalisation that alters the distribution of street_norm can push the planner onto a sequential scan, and the join goes quadratic with no error anywhere. Run EXPLAIN ANALYZE on the candidate query before adding hardware. When you do need hardware, entity resolution is memory-hungry rather than CPU-hungry and spikes during pair scoring, so a single large-RAM instance from a provider like Contabo or InterServer is often cheaper than a managed cluster you need for one stage of one job.

Common mistakes

• Tuning to a single accuracy number. Precision and recall are not interchangeable here.
• Destroying source records on merge. Every merge you cannot reverse is a merge you cannot fix.
• Blocking on a disputed field. Bedroom count and property type feel selective and are exactly what portals disagree on.
• Trusting geocode proximity alone. Two flats in one block are metres apart and are not the same property.
• Dropping the sub-unit. Discard “Flat 2” because it did not parse and you have built a machine that merges apartment buildings into single homes.
• No feedback loop. Review decisions are labelled training data arriving for free.

Best practices

• Hold out a hand-labelled sample of a few hundred pairs, weighted toward hard cases, and score every model change against it.
• Make every merge explainable at field level. If you cannot show why, you cannot defend it to the person whose listing you broke.
• Keep the review queue small enough that it is actually worked. An unread queue is auto-merging with extra steps.
• Stamp each cluster with the rule version that produced it, so changes can be evaluated rather than guessed at.
• Emit merge rate, split rate, queue depth, largest block size and per-pass contribution as first-class metrics. A managed backend such as Grafana Cloud saves running another stateful service, though self-hosted Prometheus is fine if you already have it.
• Cache photo hashes in object storage keyed by source image URL. Hashing is the slow part of every rerun and never needs doing twice. Storage without egress fees, such as Cloudflare R2, fits when hashing and storage are not colocated.

FAQ

Can I just match on address and postcode?

For one market with clean feeds and no apartment blocks, further than you would think. It breaks on sub-units, named properties, plot-marketed new-builds and portals that abbreviate differently. Treat exact address matching as one high-confidence signal, not the whole decision.

Which library should I start with?

Splink for the model, libpostal for address normalisation, your existing database for blocking. Splink’s advantage is that it explains its scores. Zingg and the Python Record Linkage Toolkit solve adjacent problems and suit teams that prefer active learning or lightweight prototyping.

Do I need machine learning for property listing deduplication?

Not to start. Deterministic rules on a good identifier plus a handful of weighted comparisons handle most pairs. The probabilistic model earns its place on the residue, which is where the expensive mistakes live. Build the deterministic layer first so you can measure what is left.

How do I handle deliberately hidden addresses?

Lean on non-address signals: photo hashes, floor area, agency identity, price. Accept that some pairs are unresolvable and design the queue to hold them rather than forcing a decision. A permanently unresolved pair beats a confidently wrong merge.

Should one property from two agencies be one record or two?

A product decision, not a technical one, and worth settling before you build. Multi-agency instructions are legitimate. Resolving to one property with several current listing events keeps both facts intact and leaves the display choice to the front end.

How often should the pipeline re-resolve?

Incrementally on every ingest, so new listings join existing clusters immediately, plus a full rebuild on a slower cadence to pick up rule changes and correct drift. The rebuild is the step people skip, and it catches clusters that were correct when formed and are not any more.

The one thing worth remembering

Property listing deduplication is not really a matching problem. It is a problem of asymmetric error costs wearing a matching problem’s clothes.

A duplicate card is visible, cheap and gets reported. A false merge is invisible, plausible, and quietly poisons everything downstream. Build your thresholds, blocking passes and survivorship rules around that asymmetry, keep the source records so mistakes stay reversible, and put a human in the band where the evidence is not conclusive. The rest is tuning.

Need help with a multi-portal listing pipeline?

This is the kind of work I do. If you are running property data from several portals and something is not adding up, I can help with:

• Designing blocking and match scoring for cross-portal feeds, including the multi-pass strategy and the metrics to watch it with.
• Auditing an existing dedupe pipeline for false merges, oversized clusters and passes that quietly stopped contributing.
• Building the two-level property and listing-event model so price history, days on market and duplicate suppression stop fighting each other.
• Setting up survivorship rules with field-level provenance, so a bad merge is a ten-minute fix rather than a restore.
• Migrating ingestion off deprecated transports onto current portal or MLS APIs without losing history.
• Right-sizing the infrastructure so the resolution job runs on something appropriate rather than whatever was already there.

Send me two records that should have matched and didn’t, or two that shouldn’t have and did, and I’ll tell you what’s going on.

The post One Home, Six Portals: Property Listing Deduplication Without False Merges appeared first on John Nessime.