Automate Internal Linking at Scale: Build a Semantic Link Recommendation Engine with Python and Embeddings
Most internal linking advice stops at “add contextual links to related posts.” That guidance is useless once you have 400 URLs, because no human is going to re-read the whole site every time they publish. The links you add by hand decay: you link the new post to three old ones, then forget that twelve older posts should now point into the new one. Link equity pools in your oldest pages and your freshest, highest-intent content sits orphaned three clicks deep.
This post walks through an internal linking engine I run as a batch job: it embeds every page on the site, computes semantic similarity between pages, filters out pairs that are already linked, and outputs a ranked list of “you should link page A to page B with this anchor” suggestions. It’s the same embeddings stack we’ve used for content cannibalization detection — pointed at a different problem. Everything here is reproducible Python; no SaaS link tool required.
Why semantic similarity beats keyword matching for internal links
The naive approach is to match on shared keywords: if two pages both contain “core web vitals,” link them. This breaks in two directions. It misses pages that are topically related but use different vocabulary (“LCP optimization” vs “page speed”), and it over-links pages that share a common word but cover unrelated subjects (“Python for log analysis” vs “Python for redirect mapping” — same language, different intent).
Embeddings solve both. A modern embedding model maps each page to a vector where semantic closeness — not lexical overlap — determines distance. Two pages about rendering JavaScript end up near each other even if one says “headless browser” and the other says “client-side hydration.” If you want the trade-offs between embedding providers, we benchmarked them in OpenAI vs Voyage vs Cohere vs BGE; for an internal-linking job on a few hundred pages, a small open-source model like BAAI/bge-small-en-v1.5 is more than enough and costs nothing to run locally.
The pipeline, end to end
The job has five stages: crawl the content, embed it, compute the similarity matrix, subtract links that already exist, and rank what’s left. Here’s the shape of it.
1. Pull the content and existing link graph
You need two things from every URL: the body text (to embed) and the set of internal links it already contains (so you don’t recommend a link that’s already there). If you’re on WordPress, the REST API gives you both without crawling HTML:
import requests, re
BASE = "https://example.com/wp-json/wp/v2"
def fetch_posts():
posts, page = [], 1
while True:
r = requests.get(f"{BASE}/posts",
params={"per_page": 100, "page": page,
"_fields": "id,link,title,content"})
if r.status_code != 200 or not r.json():
break
posts.extend(r.json())
page += 1
return posts
def extract_text(html):
return re.sub(r"<[^>]+>", " ", html)
def existing_links(html):
return set(re.findall(r'href="(https://example\.com/[^"]+)"', html))
Now each post is a record of id, link, title, cleaned text, and an outlinks set. That last field is what stops the engine from suggesting links that already exist.
2. Embed every page
Embed the title plus a truncated slice of the body. Titles carry disproportionate topical signal, so prepending the title sharpens the vector. Truncating to roughly the first 2,000 characters keeps things fast and avoids diluting the page’s main topic with footer boilerplate.
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
def embed(posts):
texts = [f"{p['title']['rendered']}. {extract_text(p['content']['rendered'])[:2000]}"
for p in posts]
return model.encode(texts, normalize_embeddings=True)
vectors = embed(posts) # shape: (n_posts, 384)
Normalizing the embeddings means cosine similarity reduces to a dot product, which makes the next step a single matrix multiply.
3. Compute pairwise similarity
sim = vectors @ vectors.T # (n, n) cosine similarity matrix
np.fill_diagonal(sim, 0) # a page is not its own link target
For a few hundred pages this is instant. If you cross into tens of thousands of URLs, swap the dense matrix for an approximate-nearest-neighbor index (FAISS or hnswlib) so you only score each page against its top candidates instead of the whole corpus.
4. Subtract the links that already exist
This is the step the manual approach can’t scale and the step that makes the output actually useful. For each candidate pair (source → target), drop it if the source already links to the target.
def candidate_links(posts, sim, threshold=0.55, max_per_page=4):
suggestions = []
for i, src in enumerate(posts):
ranked = np.argsort(sim[i])[::-1]
added = 0
for j in ranked:
if sim[i][j] < threshold:
break
target = posts[j]
if target["link"] in src["outlinks"]:
continue # already linked, skip
suggestions.append({
"from": src["link"],
"to": target["link"],
"to_title": target["title"]["rendered"],
"score": round(float(sim[i][j]), 3),
})
added += 1
if added >= max_per_page:
break
return suggestions
The threshold and max_per_page knobs are where judgment lives. A threshold around 0.55 is a reasonable starting point for BGE-small; tune it by eyeballing the first 30 suggestions. Capping at four links per page keeps you from dumping twenty links into one post and triggering the over-optimization smell that quality systems notice.
5. Generate the anchor text
Exact-match anchors repeated across the site look manufactured. The cheap, robust default is to use the target page’s title (or a shortened form of it) as the anchor, which reads naturally and varies automatically. If you want phrasing that fits the surrounding sentence, hand the source paragraph and target title to an LLM with a one-line prompt: “Suggest a 2–5 word anchor phrase, drawn from the source text, that would naturally link to the target.” That keeps anchors contextual without you writing each one.
Turning suggestions into a review queue, not an auto-publisher
Resist the urge to auto-inject these links. Even a good similarity engine produces a long tail of plausible-but-wrong pairs, and a bad internal link is worse than a missing one — it sends users and crawlers somewhere irrelevant. The pattern that works: write the ranked suggestions to a CSV or a database table, and review them in batches.
import csv
def write_queue(suggestions, path="link_suggestions.csv"):
suggestions.sort(key=lambda s: s["score"], reverse=True)
with open(path, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=["from","to","to_title","score"])
w.writeheader()
w.writerows(suggestions)
print(f"{len(suggestions)} suggestions written to {path}")
From there you can review the top-scoring rows by hand, or — if you want to close the loop — feed approved rows into a script that opens each source post via the WordPress REST API and inserts the link in the most relevant paragraph. I keep a human in the loop on approval but automate the actual edit, which is the same division of labor we used when reverse-engineering a competitor’s internal link graph: machines find the candidates, a person makes the call.
Results and what to watch
On a 300-page site, a first run of this pipeline typically surfaces 150–250 high-confidence link opportunities that don’t already exist — most of them pointing into recently published pages that the older content never got updated to reference. That’s the asymmetry hand-linking always misses: you remember to link new→old, almost never old→new.
Three things to keep an eye on. First, watch for hub pages that show up as a target for everything — if one page is suggested as a link target 80 times, your threshold is too low or that page is too generic. Second, re-run the job after every publishing batch, not on a fixed schedule, because the opportunity set changes the moment new content lands. Third, measure the outcome: track crawl depth and impressions on the pages that gained inbound links, the same way you’d monitor any change in a keyword-clustering or content workflow. If a page moves from four clicks deep to two and its impressions climb, the engine earned its runtime.
The whole thing runs in well under a minute for a few hundred pages on a laptop, with no recurring API cost if you use a local embedding model. It’s one of the highest-leverage automations you can build: link equity you already have, redistributed to the pages that need it.
Frequently asked questions
How is this different from a content cannibalization checker?
They share the embeddings step but optimize for opposite outcomes. A cannibalization checker flags pairs that are too similar because they compete for the same query. An internal-linking engine wants pairs that are related but distinct — close enough to be relevant, far enough that one isn’t a near-duplicate of the other. In practice you set an upper similarity bound too, dropping pairs above ~0.9 since those are cannibalization candidates, not link candidates.
Won’t automated internal links look spammy to Google?
Only if you auto-inject without review and over-link. The safeguards here — a per-page link cap, a similarity threshold, varied title-based anchors, and a human approving the queue — produce links that are indistinguishable from careful manual linking, because the placement logic is the same. The automation finds the opportunities; it doesn’t lower the bar for what counts as a good link.
Do I need a paid embedding API for this?
No. A local open-source model such as bge-small-en-v1.5 handles sites up to several thousand pages comfortably on CPU. Paid APIs (OpenAI, Voyage, Cohere) buy you marginally better retrieval quality and zero local setup, which matters at very large scale, but for internal linking on a typical content site the open-source model is the pragmatic default.
How often should I run the pipeline?
Run it after each batch of new content rather than on a calendar. The newest pages are the ones most likely to be orphaned and the ones most likely to deserve fresh inbound links, so triggering the job on publish gives you the highest-value suggestions exactly when they matter.
Found this useful? Bookmark SEO Automation Club and check back — we publish working automation playbooks for SEO practitioners and growth engineers every day, code included, no fluff. If you build pipelines like this, you’ll also want our breakdown of which embedding model to choose before you wire one into production.
