Automate Internal Linking at Scale: A Python + Embeddings Pipeline for Contextual Link Suggestions
Most sites have thousands of internal linking opportunities sitting idle. A page publishes, it earns a few links from whatever the author remembered to reference, and then it drifts. Six months later it is functionally an orphan on a topic three other pages could have pointed to. Doing this manually does not scale past a few hundred URLs, and the “related posts” widget your CMS ships with links on recency or category, not on semantic relevance — which is exactly the signal Google cares about.
This post walks through an internal linking pipeline that reads your whole site, embeds every page, and proposes contextual link insertions ranked by semantic similarity — with guardrails so you do not bury a page in near-duplicate anchors or link to something only loosely related. It is the same embeddings machinery we used for keyword clustering, pointed at a different problem. Everything here runs on a laptop and a few dollars of API calls.
Why internal linking is an automation problem, not an editorial one
Internal links do three jobs at once: they distribute PageRank toward the URLs you want to rank, they give Google contextual anchor text about what a page is about, and they help users (and crawlers) discover deep content. The editorial approach — asking writers to add links as they publish — fails at all three predictably. Writers link to what they remember, which biases toward recent and popular pages. Old pages never get new links pointing at them. And nobody goes back to retrofit a link into a two-year-old article when a fresh, more relevant target appears.
The automation reframe is simple: treat every page as a candidate source of links and every page as a candidate target, score every source-target pair by relevance, filter out the pairs that already link or that would be spammy, and surface the top opportunities for a human to approve. That is a ranking problem, and ranking problems are where embeddings shine.
What “relevant” means in vector space
An embedding is a fixed-length vector that encodes the meaning of a chunk of text. Two pages about the same subject land close together in that space even if they share no exact keywords. Cosine similarity between their vectors gives you a 0-to-1 relevance score. That single number replaces the brittle keyword-overlap heuristics most “related posts” plugins rely on, and it is why we lean on vector search across the site — see the vector database comparison if you want to run this over tens of thousands of URLs.
The pipeline, step by step
The flow has four stages: crawl and extract, embed, score and filter, and render suggestions. We will build each one.
1. Crawl and extract clean body text
You want the main content of each page, not the nav, footer, or sidebar — boilerplate poisons your embeddings because every page shares it. If you run WordPress, the REST API hands you clean content without scraping:
import requests, html2text
def fetch_posts(base):
posts, page = [], 1
while True:
r = requests.get(
f"{base}/wp-json/wp/v2/posts",
params={"per_page": 100, "page": page,
"_fields": "id,link,title,content,slug"},
timeout=30)
if r.status_code != 200 or not r.json():
break
posts.extend(r.json())
page += 1
return posts
h = html2text.HTML2Text(); h.ignore_links = True; h.ignore_images = True
pages = [{
"id": p["id"], "url": p["link"], "slug": p["slug"],
"title": p["title"]["rendered"],
"text": h.handle(p["content"]["rendered"])[:6000],
} for p in fetch_posts("https://example.com")]For non-WordPress sites, swap in a real crawler. We covered the trade-offs in the SEO crawler comparison — any of them can export URLs plus rendered HTML that you feed through the same extraction step. Cap the text length: the first ~6,000 characters carry the page’s topic, and trimming keeps embedding costs and token limits sane.
2. Embed every page
Two realistic choices. A hosted model (OpenAI’s text-embedding-3-small at 1536 dimensions) is cheapest to operate and gives strong results for a few cents across a whole site. A local model (sentence-transformers with bge-small-en-v1.5) costs nothing per call and keeps data on your machine, at the price of running the model yourself. The pipeline is identical either way — you get one vector per page.
from openai import OpenAI
import numpy as np
client = OpenAI()
def embed(texts):
resp = client.embeddings.create(
model="text-embedding-3-small", input=texts)
return np.array([d.embedding for d in resp.data])
# Embed the title + body so the vector reflects the page topic
inputs = [f"{p['title']}\n\n{p['text']}" for p in pages]
vectors = np.vstack([embed(inputs[i:i+100])
for i in range(0, len(inputs), 100)])
# L2-normalize so dot product == cosine similarity
vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)3. Score every pair and filter
With normalized vectors, the full similarity matrix is a single matrix multiply. The interesting work is in the filters — this is where you stop the pipeline from producing garbage.
import re
sim = vectors @ vectors.T # N x N cosine similarities
np.fill_diagonal(sim, 0) # a page never links to itself
def already_links(src, dst):
return dst["url"] in src_html[src["id"]] # raw HTML lookup
SIM_FLOOR = 0.55 # below this, "related" is noise
MAX_OUT_NEW = 5 # cap new links added per source page
suggestions = []
for i, src in enumerate(pages):
ranked = np.argsort(sim[i])[::-1]
added = 0
for j in ranked:
if sim[i][j] < SIM_FLOOR or added >= MAX_OUT_NEW:
break
dst = pages[j]
if already_links(src, dst):
continue
anchor = find_anchor(src["text"], dst["title"])
if not anchor:
continue # no natural anchor phrase in the source body
suggestions.append({
"source": src["url"], "target": dst["url"],
"score": round(float(sim[i][j]), 3), "anchor": anchor})
added += 1Three guardrails are doing the heavy lifting. The similarity floor (0.55 is a sane starting point; tune it against a sample you eyeball) keeps you from linking pages that merely share a vocabulary. The out-link cap stops a hub page from sprouting forty new links in one pass, which reads as spam to both users and Google. And the anchor requirement — find_anchor looks for a phrase in the source body that plausibly matches the target’s topic — means you only suggest a link where a natural insertion point already exists, rather than forcing exact-match anchors, which is the fastest way to earn an over-optimization footprint.
4. Render an approval queue, not an auto-publisher
Resist the urge to write links back automatically on day one. Dump the ranked suggestions to a CSV or a Looker Studio table and have a human approve the first few batches. You will catch systematic errors — a tag page ranking as a target, a similarity floor set too low — before they touch production. Once the precision looks good on a sample, you can promote approved rows to an automated writer that patches the post content through the same REST API.
import csv
suggestions.sort(key=lambda s: s["score"], reverse=True)
with open("internal_link_suggestions.csv", "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=suggestions[0].keys())
w.writeheader(); w.writerows(suggestions)Wiring it into a recurring workflow
A one-off run is useful; a scheduled one compounds. The natural shape is a weekly job: re-crawl, re-embed only the pages that changed since last run (cache vectors by content hash so you are not re-embedding the whole site), regenerate suggestions, and drop the new opportunities into a review queue. If you already orchestrate SEO jobs in n8n, this slots in as a scheduled workflow that calls your Python step and posts the top suggestions to Slack. We compared the orchestration options in the n8n vs Make vs Zapier teardown if you have not picked a runner yet.
Pair this with orphan detection and the two close a loop: orphan detection tells you which pages have no inbound links, and this pipeline tells you where to add them and with what anchor. If you have not run the first half, start with automated orphan page detection and feed its output in as a priority list of targets.
Results and what to watch
On a mid-size content site (roughly 400 posts), a first pass typically surfaces several hundred candidate links, of which a human approves 30–40% after the filters above — the rest are near-duplicates, already-linked pairs, or targets without a clean anchor. The pages that benefit most are the deep, older articles that never earned links naturally: giving each of them three or four new contextual inbound links from relevant sources is exactly the internal PageRank flow that moves striking-distance pages.
Two failure modes to watch. First, topic drift toward hubs: your highest-similarity targets will cluster around a few broad pages, so the out-link cap and a per-target inbound cap matter as much as the floor. Second, anchor monotony: if every suggestion for a target uses the identical anchor phrase, vary it — anchor diversity is a signal, and identical anchors site-wide look manufactured. Track the distribution of anchors per target and flag any that exceed a threshold.
Measure the outcome, not just the output. Tag the batch of edited URLs, then watch impressions and average position for the target pages in Search Console over the following weeks. Internal linking effects are gradual and confounded by everything else you ship, so if you want a clean read, run it as a proper experiment — the approach in our SEO split testing guide applies directly.
Takeaways
Internal linking at scale is a ranking-and-filtering problem, and embeddings turn “which pages are related?” into a single cosine score you can sort. The value is not the similarity matrix — that is the easy part — it is the guardrails: a similarity floor, out-link and inbound caps, an already-linked check, and a natural-anchor requirement. Ship it as an approval queue first, measure the targets in GSC, and only then let it write links unattended. Do that and you convert a chore nobody does into a weekly job that keeps your best-buried pages linked.
Found this useful? Bookmark SEO Automation Club and check back — we publish working automation playbooks like this one every week, code included. If you are building out your embeddings stack, the vector database comparison is the natural next read.
Frequently asked questions
What similarity threshold should I use for internal link suggestions?
Start around 0.55 cosine similarity with L2-normalized embeddings, then tune against a hand-checked sample. Too low and you link loosely related pages; too high and you only ever suggest near-duplicates. The right floor depends on your embedding model and how tightly focused your site’s topics are, so calibrate rather than copying a number.
Should I let the pipeline insert links automatically?
Not at first. Run it as an approval queue for the first several batches so you can catch systematic errors — tag pages ranking as targets, a floor set too low, or monotonous anchors. Once precision on a sample looks reliably good, you can promote approved suggestions to an automated writer that patches post content through the CMS API.
Do I need a vector database for this?
For a few hundred to a few thousand pages, no — a NumPy array in memory and a single matrix multiply is faster and simpler. A vector database earns its keep at tens of thousands of URLs or when you need incremental updates and shared access across jobs.
Which embedding model is best for SEO internal linking?
Any strong general-purpose text embedding model works. A hosted option like text-embedding-3-small is cheap and low-maintenance; a local sentence-transformers model such as bge-small-en-v1.5 costs nothing per call and keeps data on your machine. Relevance quality is similar for this task, so choose based on cost, privacy, and operational preference.
