Detect Thin and Doorway Pages at Scale: An LLM Quality-Gate Pipeline in Python

Most site owners discover their thin-content problem the same way: a Helpful Content update lands, traffic slides across a whole section at once, and it becomes obvious that hundreds of near-empty pages were quietly dragging down the domain. Fixing that by hand does not scale. What scales is a quality gate — an automated pipeline that inspects every URL, separates genuinely thin and doorway pages from thin-looking pages that actually rank, and hands you a defensible action for each one.

This article builds that gate in Python. The design principle is deliberate: cheap deterministic signals do the heavy filtering first, and an LLM adjudicates only the ambiguous minority. That order keeps the cost bounded and the results reproducible.

Thin vs doorway: two different problems, two different fixes

People use “thin” and “doorway” interchangeably, but Google treats them differently, and so should your pipeline.

Thin pages

A thin page has little unique value for the reader: a paragraph of filler, a scraped definition, an auto-generated stub, or a tag archive with two posts. The fix is usually improve or consolidate — merge overlapping stubs into one strong page, or expand the page until it earns its spot.

Doorway pages

A doorway is a page that exists to catch a keyword variant and funnel users somewhere else — think “plumber in [city]” replicated across 400 near-identical templates. The tell is structural near-duplication at scale plus a shared destination. The fix is almost always consolidate or remove, because improving 400 clones individually is neither possible nor desirable.

Because the remedies diverge, the detector cannot output a single “bad page” flag. It needs to distinguish low-value-standalone from part-of-a-duplicate-cluster.

Architecture: a deterministic pre-filter, then an LLM judge

It is tempting to loop every URL through a large model and ask “is this thin?” Do not. On a 5,000-URL site that is 5,000 API calls per audit, it is non-deterministic between runs, and it spends tokens on pages that a word counter could clear in microseconds.

Instead, use two stages:

1. Stage 1 — structural signals (free, deterministic): word count, main-content ratio, template/boilerplate share, near-duplicate clustering, internal-link posture, and search performance. This alone resolves the clear-cut majority.
2. Stage 2 — LLM adjudication (paid, targeted): only the borderline pages that survive Stage 1 get an LLM quality judgment, returned as structured JSON.

If you have already built an LLM content-quality scorer aligned to the Quality Rater Guidelines, Stage 2 here is a focused, cheaper cousin of it: fewer questions, a narrower verdict.

Stage 1: structural signals in Python

Start by extracting the main content — not the whole DOM. Boilerplate (nav, footer, sidebars) inflates word counts and hides thinness, so measure the ratio of main content to total text.

import trafilatura
from bs4 import BeautifulSoup

def content_signals(html: str) -> dict:
    full_text = BeautifulSoup(html, "html.parser").get_text(" ", strip=True)
    main = trafilatura.extract(html) or ""
    full_words = max(len(full_text.split()), 1)
    main_words = len(main.split())
    return {
        "main_words": main_words,
        "main_ratio": round(main_words / full_words, 3),  # low = boilerplate-heavy
        "main_text": main,
    }

A page with 900 total words but a 0.12 main ratio is mostly chrome. That single ratio catches a lot of “looks full, reads empty” templates.

Near-duplicate clustering for doorways

Doorways reveal themselves as clusters. Use MinHash + LSH so you can compare thousands of pages without an O(n²) blowup.

from datasketch import MinHash, MinHashLSH

def shingles(text, k=5):
    tokens = text.lower().split()
    return {" ".join(tokens[i:i+k]) for i in range(len(tokens) - k + 1)}

def build_lsh(pages, threshold=0.8):
    lsh = MinHashLSH(threshold=threshold, num_perm=128)
    minhashes = {}
    for url, text in pages.items():
        m = MinHash(num_perm=128)
        for sh in shingles(text):
            m.update(sh.encode("utf8"))
        lsh.insert(url, m)
        minhashes[url] = m
    clusters = {u: lsh.query(m) for u, m in minhashes.items()}
    return clusters  # url -> list of near-duplicate urls

Any URL whose cluster has, say, five or more members is a strong doorway candidate — especially if those members all link to the same money page. That destination overlap is the signal that separates a legitimate content series from a doorway farm.

Join with performance and link data

Thinness only matters if the page also fails to earn anything. Pull impressions and clicks from Search Console and internal-link counts from your crawl, then join everything on URL. A short page with real impressions is not your problem; a short, orphaned, zero-impression page is. The link side pairs naturally with an orphan-page detection pipeline, since orphaned and thin is the worst quadrant.

import pandas as pd

def score_row(r):
    reasons = []
    if r.main_words < 300: reasons.append("very_short")
    if r.main_ratio < 0.35: reasons.append("boilerplate_heavy")
    if r.cluster_size >= 5: reasons.append("near_duplicate_cluster")
    if r.impressions_90d == 0 and r.inlinks <= 1: reasons.append("no_demand_no_links")
    return reasons

df["reasons"] = df.apply(score_row, axis=1)
df["needs_llm"] = df.reasons.apply(lambda x: 0 < len(x) < 3)  # ambiguous middle

Pages with zero reasons pass. Pages with three or more reasons are near-certain and can skip straight to a recommended action. Only the ambiguous middle — needs_llm — proceeds to Stage 2.

Stage 2: LLM adjudication with structured output

For the survivors, ask the model a tight question and force a machine-readable answer. Do not ask “is this good?” — ask for a verdict plus the single most useful next action.

import json
from openai import OpenAI

client = OpenAI()

PROMPT = """You are a strict SEO content reviewer.
Given the main content of one page, decide if it provides
independent value to a searcher or is thin/doorway.
Return JSON only:
{"verdict":"valuable|thin|doorway",
 "confidence":0-1,
 "action":"keep|improve|consolidate|noindex",
 "why":"one sentence"}"""

def adjudicate(main_text: str) -> dict:
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": PROMPT},
            {"role": "user", "content": main_text[:6000]},
        ],
    )
    return json.loads(r.choices[0].message.content)

Because Stage 1 already removed the obvious cases, this call runs on a small fraction of the site. On a 5,000-URL audit you might send only 300–600 pages to the model, which is the difference between a few cents and a real bill.

From scores to actions

Collapse both stages into one decision column. A simple, auditable policy works better than a black box:

• keep — passed Stage 1, or LLM says valuable with high confidence.
• improve — thin but has impressions or backlinks worth saving; expand and add unique depth.
• consolidate — member of a near-duplicate cluster; 301 the clones into one canonical page.
• noindex — no demand, no links, no unique value; remove from the index (and consider whether it belongs in the crawl at all — the same logic drives pagination and faceted-navigation audits).

Export the table, sort by potential impact (impressions at risk × cluster size), and work top-down. Never let the pipeline act on its own: it proposes, a human confirms the noindex and consolidation batches, and only then do you push changes.

Guardrails against false positives

A thin detector that is too eager will noindex pages you want. Three cheap protections:

First, respect demand: never auto-noindex a page with meaningful impressions, however short. Short can be correct — a definition, a status page, a tool. Second, respect intent: some queries are satisfied by 150 words, and a page that ranks for those is doing its job. Third, whitelist templates: product pages, contact pages, and legal pages are structurally repetitive by design and should be excluded from the doorway clustering. Encode these as hard exemptions before the scoring runs, not as afterthoughts.

Frequently asked questions

How is a doorway page different from a legitimate location or category page?

Intent and independence. A legitimate location page answers a real local query with unique information — hours, staff, inventory, directions. A doorway is a near-duplicate template whose only purpose is to capture a keyword variant and route the user elsewhere. The near-duplicate clustering plus shared-destination check is what separates them programmatically.

What word count makes a page “thin”?

There is no magic number, and word count alone is a weak signal. A 250-word page that fully answers a query is fine; a 1,200-word page of filler is not. Use word count only as one input alongside main-content ratio, demand, and uniqueness — never as the sole gate.

Do I need an LLM at all for this?

For clear cases, no — structural signals resolve most of a site. The LLM earns its cost only on the ambiguous middle, where a page is short but not obviously worthless. Running the model on every URL is the expensive, non-reproducible anti-pattern this pipeline is designed to avoid.

Should thin pages be deleted, noindexed, or consolidated?

It depends on value and duplication. Consolidate near-duplicates with a 301 to a canonical page; noindex low-value pages that have no demand and no links; improve pages that are thin but attract impressions or backlinks. Deletion (410) is reserved for pages with no value and no equity to preserve.

How often should the quality gate run?

Monthly for most sites, or after any large programmatic publish. Content decays and new templates ship, so treat the gate as a recurring audit rather than a one-time cleanup, and diff each run against the last to catch regressions early.

Build it once and the payoff compounds: instead of waiting for the next algorithm update to expose your thin pages, you find and fix them on your own schedule, with an audit trail behind every decision.