Detect Near-Duplicate Pages at Scale with MinHash and LSH: The Python Pipeline Embeddings Get Wrong

If your site has more than a few thousand URLs, you almost certainly have near-duplicate pages you can’t see: paginated archives that echo each other, product variants with 95% identical copy, tag pages that recycle the same intro, printer-friendly versions, syndicated posts, and boilerplate-heavy templates where the unique content is a thin sliver in a sea of shared markup. Google notices these long before you do. They dilute crawl budget, split link equity, and — since the Helpful Content system matured — quietly drag down the perceived quality of the whole domain.

The reflexive 2026 answer is “use embeddings.” It’s the wrong tool for this specific job. Embeddings measure semantic similarity: two pages about the same topic land close together even when they share zero sentences. That’s exactly what you want for keyword cannibalization detection, where the problem is two different articles competing for one query. It’s exactly what you don’t want for near-duplicate detection, where the problem is two pages that literally share most of their text. This article covers the tool built for textual near-duplication — MinHash with Locality-Sensitive Hashing (LSH) — why it scales where pairwise comparison and embeddings both choke, and how to wire it into a decision pipeline that tells you what to consolidate, canonicalize, or prune.

Why embeddings and brute force both fail here

Start with the naive approach: compare every page to every other page. With n URLs that’s n(n−1)/2 comparisons — for a 50,000-page site, about 1.25 billion. Even at a microsecond each, that’s 20+ minutes of pure comparison before you’ve done anything useful, and it explodes quadratically as the site grows. Brute force is a non-starter above a few thousand URLs.

Embeddings don’t solve the scaling problem and they answer the wrong question. Cosine similarity between two embedding vectors tells you the pages mean similar things. But a near-duplicate isn’t defined by meaning — it’s defined by shared strings. Consider two cases:

• Two independently written 1,500-word guides on “how to file taxes in Chile.” High semantic similarity, low textual overlap. Not a near-duplicate — this is legitimate topical overlap (or cannibalization), and consolidating them would destroy real content.
• A product page and its “?color=blue” variant that differ by one word. Near-identical text, and the embedding model may actually score them slightly lower than the tax guides because the copy is short and generic. This is the near-duplicate that wastes crawl budget.

Semantic similarity and textual near-duplication are different axes. MinHash estimates Jaccard similarity — the ratio of shared tokens to total unique tokens — which is precisely the “how much text do these two pages literally share” question. That’s the axis that predicts whether Google will fold two URLs into one cluster and pick a canonical for you.

The shingling → MinHash → LSH pipeline

Step 1: Shingle each document

A shingle (or w-gram) is a sliding window of w consecutive tokens. The document “the quick brown fox” with w=3 becomes {“the quick brown”, “quick brown fox”}. Shingling converts each page into a set, and the Jaccard similarity of two shingle sets is a robust measure of textual overlap that’s insensitive to small edits and word reordering. Word-level shingles of 5–9 tokens work well for editorial content; shorter shingles over-match on common phrases, longer ones miss paraphrases. Critically, strip your boilerplate first — nav, footer, sidebar — or every page will look like a near-duplicate of every other page on your own template.

import re

def shingle(text: str, w: int = 7) -> set[str]:
    tokens = re.findall(r"\w+", text.lower())
    if len(tokens) < w:
        return {" ".join(tokens)}
    return {" ".join(tokens[i:i+w]) for i in range(len(tokens) - w + 1)}

Step 2: MinHash the shingle sets

Storing full shingle sets for 50,000 pages is expensive, and comparing them is slow. MinHash compresses each set into a fixed-length signature — say 128 integers — with a remarkable property: the fraction of positions where two signatures agree is an unbiased estimator of their true Jaccard similarity. You trade a tiny, tunable amount of accuracy for a massive reduction in storage and comparison cost. The datasketch library implements this cleanly:

from datasketch import MinHash, MinHashLSH

def make_minhash(shingles: set[str], num_perm: int = 128) -> MinHash:
    m = MinHash(num_perm=num_perm)
    for s in shingles:
        m.update(s.encode("utf8"))
    return m

More permutations means a tighter estimate (standard error is roughly 1/√num_perm) at higher cost. 128 gives an error around 9%; 256 halves it. For near-duplicate triage, 128 is plenty.

Step 3: LSH to avoid the quadratic wall

This is the step that makes the whole thing scale. LSH splits each 128-integer signature into b bands of r rows each (b × r = 128). Two documents become candidates only if they hash identically in at least one band. Similar documents share many bands and collide often; dissimilar ones almost never do. You go from comparing all pairs to comparing only the handful that land in the same bucket — effectively linear time.

lsh = MinHashLSH(threshold=0.7, num_perm=128)
signatures = {}

for url, text in pages.items():          # pages: dict[url -> cleaned body text]
    m = make_minhash(shingle(text))
    signatures[url] = m
    lsh.insert(url, m)

# Query near-duplicates for each page
seen, clusters = set(), []
for url, m in signatures.items():
    if url in seen:
        continue
    dupes = lsh.query(m)                  # returns candidate near-duplicates
    if len(dupes) > 1:
        clusters.append(dupes)
        seen.update(dupes)

The threshold parameter tunes the band/row split so the collision probability curve is steepest around your target similarity. Setting it to 0.7 means “surface pairs that are roughly 70%+ textually identical.” That threshold is a policy decision, not a default — the next section is about choosing it deliberately.

Turning clusters into decisions

A list of near-duplicate clusters is not an action plan. The mistake teams make is treating every cluster the same way — bulk-canonicalizing or bulk-noindexing — when the right move depends on similarity strength and on which page earns traffic. Join your clusters against Search Console data and route each one:

• Similarity 0.9–1.0, one page gets all the clicks: true duplicates. Canonicalize the dead variants to the winner, or 301 them. Query variants, session IDs, and print versions live here.
• Similarity 0.9–1.0, clicks split across the cluster: you’re cannibalizing yourself with duplicate content. Consolidate into one authoritative URL and redirect the rest — the same logic behind a disciplined redirect-mapping workflow.
• Similarity 0.7–0.9, all pages thin: templated thin content (think 200-word location pages with one town name swapped). Either merge into a substantial hub or noindex the lot. This is the category the Helpful Content system punishes hardest.
• Similarity 0.7–0.9, pages have distinct value: false alarm from shared boilerplate that survived your extraction. Improve your body-text extraction and re-run rather than editing content.

The traffic join is what separates a useful audit from a dangerous one. Never noindex or redirect a page that’s earning impressions without checking what you’d lose. And feed the surviving URLs back into your sitemap and index-bloat audit so the pages you pruned actually leave the index instead of lingering as soft-404s.

Tuning: the two knobs that matter

Two parameters do the heavy lifting, and getting them wrong produces either noise or silence.

Shingle width (w). Too small and unrelated pages collide on stock phrases like “in this article we will”; too large and a modest rewrite hides a genuine duplicate. For prose, start at w=7. For structured/short content like product descriptions, drop to w=4–5 so you still capture overlap in brief text.

LSH threshold. This trades precision against recall. A high threshold (0.85) surfaces only near-identical pages — few false positives, but paraphrased duplicates slip through. A low threshold (0.6) casts a wide net and will flag pages that merely share a template. Run it at two thresholds and treat the high-confidence band as auto-actionable and the low-confidence band as a review queue. Validate by hand-checking 20–30 flagged pairs before you trust any threshold on a new site — the “right” value depends on how much boilerplate your CMS injects, which no default can know.

Where this fits in an automated SEO stack

Near-duplicate detection is cheap enough to run weekly. The full pipeline — crawl, extract body text, shingle, MinHash, LSH-cluster, join to GSC, emit a routed action report — runs in minutes on a mid-size site because LSH keeps it linear. Schedule it, diff each run against the last so you catch new duplication as it’s published rather than months later, and route high-similarity clusters straight into your CMS as canonical or redirect suggestions for human approval. The result is a domain that stays lean without anyone manually hunting for the copies — which is the entire point of treating SEO as an engineering problem rather than a checklist.

Frequently asked questions

Is MinHash better than embeddings for finding duplicate content?

For textual near-duplicates — pages that literally share most of their words — yes. MinHash estimates Jaccard similarity, which measures shared text directly, and LSH makes it scale to hundreds of thousands of URLs in near-linear time. Embeddings measure semantic similarity, which is the right tool for detecting topically overlapping but differently written pages (cannibalization), not for catching variants, boilerplate, and templated thin content.

What Jaccard similarity threshold counts as a duplicate for SEO?

There’s no universal number, but a practical starting split is: 0.9+ is a true duplicate to canonicalize or redirect, 0.7–0.9 is templated or thin content to review and likely consolidate, and below 0.7 is usually legitimate overlap. Always validate the threshold by manually inspecting 20–30 flagged pairs on your specific site before acting in bulk, because CMS boilerplate inflates apparent similarity.

How many hash permutations should I use for MinHash?

128 permutations give a standard error around 9% on the similarity estimate, which is fine for triage. Use 256 if you need tighter estimates for auto-actioning decisions. More permutations improve accuracy at a linear cost in time and memory, so scale them to how much you trust the pipeline to act without review.

Do I need to remove navigation and footer text before hashing?

Yes — this is the most common cause of false positives. If you shingle the full HTML, every page shares your header, nav, sidebar, and footer, so everything looks like a near-duplicate of everything else. Extract the main body content first (readability-style extraction or a content-area selector) so MinHash compares the parts that actually differ.