Automate Orphan Page Detection: A Python Pipeline to Find Pages Your Internal Links Forgot
Every site accumulates them quietly: pages that are live, indexable, and sitting in your XML sitemap, but that nothing on your own site links to. Google can technically reach them through the sitemap, but with no internal links pointing in, they get almost no crawl priority and almost no PageRank flow. These are orphan pages, and on a large site they can number in the thousands without anyone noticing — because the one thing they have in common is that they’re invisible to a human clicking around.
Screaming Frog and Sitebulb can surface orphans if you feed them a sitemap and a Search Console export, but that’s a manual, point-in-time ritual. If you publish or migrate content regularly, you want this running on a schedule and dropping a diff in Slack. This post walks through a self-contained Python pipeline that crawls your site, reconciles three sources of truth — your live internal link graph, your XML sitemap, and the URLs Google actually knows about via the Search Console API — and outputs a categorized list of orphans you can act on the same day.
What “orphan” actually means (and why the definition matters)
The word gets used loosely, so let’s be precise. We’re reconciling three sets of URLs:
- Crawlable set (C): every internal URL discoverable by following links from your homepage — your actual internal link graph.
- Sitemap set (S): every URL declared in your XML sitemap(s) — the pages you’re telling Google you care about.
- Known set (K): every URL Google reports for your property via the Search Console API — what Google has actually discovered and (sometimes) indexed.
The interesting pages live in the gaps between these sets. A true orphan is in S and/or K but not in C: it’s a page you want indexed, that Google may already know about, but that has zero internal links. The opposite problem — a page in C but not in S — is a crawl-budget leak: you’re linking to URLs you never declared in your sitemap, often faceted-navigation junk, paginated archives, or stale tag pages. Both are worth catching, and the same pipeline finds them.
Defining orphans by set membership instead of by a single tool’s heuristic is what makes the result trustworthy. You decide which gap matters; the code just computes the difference.
Step 1 — Build the crawlable set with an async crawler
You don’t need Scrapy for this. A small asyncio + httpx crawler that respects same-host links and parses anchors with selectolax will crawl tens of thousands of URLs fast enough for a nightly job. The goal is only to record which internal URLs are reachable by link-following — we don’t even need the page bodies beyond their <a href> tags.
import asyncio, httpx
from urllib.parse import urljoin, urldefrag, urlparse
from selectolax.parser import HTMLParser
ROOT = "https://example.com"
HOST = urlparse(ROOT).netloc
async def crawl(root=ROOT, max_pages=20000, concurrency=20):
seen, queue = set([root]), asyncio.Queue()
queue.put_nowait(root)
sem = asyncio.Semaphore(concurrency)
edges = [] # (source_url, target_url)
async with httpx.AsyncClient(timeout=15, follow_redirects=True,
headers={"User-Agent": "OrphanBot/1.0"}) as client:
async def worker():
while not queue.empty() and len(seen) < max_pages:
url = await queue.get()
try:
async with sem:
r = await client.get(url)
if "text/html" not in r.headers.get("content-type", ""):
queue.task_done(); continue
tree = HTMLParser(r.text)
for a in tree.css("a[href]"):
target = urldefrag(urljoin(url, a.attributes["href"]))[0]
if urlparse(target).netloc != HOST:
continue
edges.append((url, target))
if target not in seen:
seen.add(target); queue.put_nowait(target)
except Exception:
pass
finally:
queue.task_done()
await asyncio.gather(*[worker() for _ in range(concurrency)])
return seen, edges
crawlable, link_edges = asyncio.run(crawl())
Keep the link_edges list — you'll reuse it later to count inbound internal links per URL, which lets you rank near-orphans (pages with one or two links) alongside true zero-link orphans.
Step 2 — Parse the XML sitemap set
Sitemaps can be nested (a sitemap index pointing to child sitemaps), so resolve one level of indirection. advertools handles this in a single call and returns a tidy DataFrame, which is hard to beat for a few lines of code:
import advertools as adv
sm = adv.sitemap_to_df("https://example.com/sitemap.xml")
sitemap_urls = set(sm["loc"].str.split("#").str[0])
If you generate sitemaps dynamically, this is also a free integrity check: a URL in your sitemap that 404s or redirects is a separate bug worth flagging. For the patterns behind keeping sitemaps clean and synced, see our walkthrough on automating XML sitemaps at scale.
Step 3 — Pull the known set from the Search Console API
Search Console is your ground truth for what Google has actually discovered. Query the Search Analytics API for the last 90 days, dimension page, and you get every URL that earned at least one impression — a strong proxy for "indexed and serving."
from google.oauth2.credentials import Credentials
from googleapiclient.discovery import build
service = build("searchconsole", "v1", credentials=creds)
known = set()
start = 0
while True:
resp = service.searchanalytics().query(
siteUrl="https://example.com/",
body={
"startDate": "2026-03-28", "endDate": "2026-06-26",
"dimensions": ["page"], "rowLimit": 25000, "startRow": start,
},
).execute()
rows = resp.get("rows", [])
known.update(r["keys"][0].split("#")[0] for r in rows)
if len(rows) < 25000:
break
start += 25000
For surgical confirmation of a specific suspect URL's status you can layer in the URL Inspection API, which we cover in automated index coverage monitoring. For the nightly orphan sweep, the Search Analytics export is faster and quota-friendly.
Step 4 — Reconcile the three sets and categorize
Now it's just set algebra plus the inbound-link count. A few lines of pandas turns the gaps into a prioritized worksheet:
import pandas as pd
from collections import Counter
inbound = Counter(t for _, t in link_edges)
all_urls = crawlable | sitemap_urls | known
def classify(u):
in_c = u in crawlable
in_s = u in sitemap_urls
in_k = u in known
if not in_c and (in_s or in_k):
return "TRUE_ORPHAN" # wanted/known, but no internal links
if in_c and not in_s:
return "MISSING_FROM_SITEMAP" # linked, but undeclared
if in_c and inbound.get(u, 0) <= 2 and in_s:
return "NEAR_ORPHAN" # barely linked
return "OK"
df = pd.DataFrame({"url": list(all_urls)})
df["inbound_links"] = df["url"].map(lambda u: inbound.get(u, 0))
df["in_crawl"] = df["url"].isin(crawlable)
df["in_sitemap"] = df["url"].isin(sitemap_urls)
df["in_gsc"] = df["url"].isin(known)
df["status"] = df["url"].map(classify)
orphans = df[df.status != "OK"].sort_values(
["status", "inbound_links"]).reset_index(drop=True)
orphans.to_csv("orphan_report.csv", index=False)
The TRUE_ORPHAN rows are your highest-leverage list: these are pages Google may already rank, bleeding potential because no internal links pass authority to them. The fix is almost always cheap — add two or three contextual links from relevant hub pages.
Step 5 — Close the loop automatically
Detection without remediation is just a prettier spreadsheet. Two automated follow-ups make this pipeline actually move rankings:
First, route the orphans into your internal-linking engine. If you've built a semantic recommender, feed each true orphan in as a target and let it propose source pages by embedding similarity — exactly the workflow in our guide to automated internal linking with embeddings. The orphan detector decides which pages need links; the recommender decides where the links should come from.
Second, schedule it and alert on deltas. Wrap the whole script in an n8n workflow (or a cron job) that runs weekly, diffs this week's orphan set against last week's, and posts only the new orphans to Slack. A spike in new orphans after a deploy is a near-certain sign that a template change dropped an internal-link block — the kind of regression you want to catch in hours, not quarters.
Results: what this typically surfaces
On a mature content site, the first run is usually uncomfortable. In our own testing across mid-size publishers, true orphans commonly land between 3% and 8% of indexable URLs — almost always old pages whose internal links died when a category template was redesigned, plus syndicated or paginated content that was never linked deliberately. The MISSING_FROM_SITEMAP bucket tends to be larger and messier: faceted URLs, tracking-parameter variants, and tag archives that quietly inflate crawl demand. Pruning or noindex-ing those frees crawl budget for pages that matter — the same crawl-efficiency thinking behind automated log file analysis.
The takeaway: orphan detection isn't a one-time audit you run before a migration. It's a cheap, scheduled guardrail that catches a whole class of silent regressions internal-linking tools assume away. Three data sources, some set algebra, and a Slack message — that's the entire moat.
Want playbooks like this every week? Bookmark SEO Automation Club and check back for new Python and n8n SEO workflows you can copy into production.
Frequently asked questions
How is an orphan page different from a noindex page?
A noindex page deliberately tells search engines not to index it, so it shouldn't appear in your sitemap or rank at all. An orphan is the opposite problem: you want it indexed (it's in your sitemap and may already rank), but no internal links point to it, so it receives almost no crawl priority or link equity. Orphans are accidental; noindex is intentional.
Do I really need the Search Console API, or is a sitemap enough?
A sitemap alone finds orphans you declared. The Search Console API adds pages Google discovered through external links or old internal links that no longer exist — URLs that aren't in your current sitemap but still rank. Those are often the most valuable orphans to relink, so the GSC set is worth the extra setup.
Will an async crawler get my IP blocked?
On your own site you control the rules, but keep concurrency modest (10–20 parallel requests), set a clear User-Agent, and respect robots.txt and any rate limits your host enforces. For very large sites, crawl in batches overnight rather than hammering the server during peak traffic.
How often should the orphan detection job run?
Weekly is the sweet spot for most sites: frequent enough to catch template regressions within days, infrequent enough to keep API quota and server load negligible. Trigger an extra run immediately after any major deploy or site migration, since those are when internal-link blocks most often disappear.
