Automate Redirect Mapping for Site Migrations: A Python + Embeddings Pipeline That Matches Old URLs to New
Site migrations are where organic traffic goes to die. Industry post-mortems routinely blame 20–40% traffic losses on one artifact: the redirect map. And the way most teams build that artifact has not changed in fifteen years — someone exports two crawls into a spreadsheet, sorts both columns, and eyeballs VLOOKUP results for a weekend. On a 500-URL brochure site, that is tedious but survivable. On a 40,000-URL ecommerce replatform, it is where mistakes hide: URLs mapped to the wrong template, entire directories bulk-pointed at the homepage, and long-tail pages silently dropped because nobody scrolled past row 8,000.
This post walks through a three-stage Python pipeline that builds the redirect map for you: exact rules first, fuzzy slug matching second, and content embeddings for everything that survives both. Each stage only handles what the previous one could not, every mapping carries a confidence score, and humans only review the genuinely ambiguous rows. We run a variant of this pipeline on our own projects; the numbers below come from a real 12,400-URL migration.
Why redirect mapping resists simple automation
If old and new URLs differed only by domain or a path prefix, a single rewrite rule would do. Real migrations are messier for three reasons:
Information architecture changes ship together with the migration. Teams rarely replatform without also restructuring: /blog/2019/05/post-title/ becomes /resources/post-title/, category paths get flattened, and product URLs swap SKU-based slugs for descriptive ones. The mapping between old and new is a business decision, not a string transformation.
Content gets merged, split, and deleted. A migration is usually also a content audit. Four thin service pages become one strong page; one bloated guide becomes three. A one-to-one matcher cannot represent that — you need many-to-one mappings and an explicit “gone, return 410” bucket.
The cost of errors is asymmetric and delayed. A wrong internal link costs you a click. A wrong redirect quietly bleeds equity for months, and Google treats mass redirects to irrelevant pages (the classic “everything to homepage” move) as soft 404s, meaning you paid the engineering cost of redirects and got none of the equity transfer.
So the goal is not full automation. The goal is a pipeline that resolves the 85–90% of URLs that have an unambiguous best match, and produces a short, prioritized review queue for the rest.
Pipeline architecture: three stages, one confidence score
The pipeline takes two inputs: a crawl of the legacy site (URL, title, H1, main content, and — critically — GSC clicks and referring domains per URL) and the list of new URLs with the same fields from staging. It emits one row per legacy URL: old_url, new_url, method, confidence, traffic_value.
Stage 1: deterministic rules
Before any fuzzy logic, harvest the free wins. Most migrations have systematic transformations that map thousands of URLs with zero ambiguity — protocol and host changes, dropped date folders, swapped category prefixes. Express them as ordered regex rules and validate each candidate against the real new URL set:
import re
RULES = [
(r"^/blog/\d{4}/\d{2}/([^/]+)/?$", r"/resources/\1/"),
(r"^/products/([a-z0-9-]+)-\d{5,}/?$", r"/shop/\1/"),
(r"^/category/(.+)$", r"/c/\1"),
]
def apply_rules(old_path, new_url_set):
for pattern, template in RULES:
candidate = re.sub(pattern, template, old_path)
if candidate != old_path and candidate in new_url_set:
return candidate # confidence 1.0
return NoneThe candidate in new_url_set check is the part spreadsheet workflows skip: a rule that produces a URL that does not actually exist on staging is a future 404 chain, not a redirect. On our 12,400-URL migration, eleven rules resolved 7,900 URLs (64%) at confidence 1.0.
Stage 2: fuzzy slug matching
For URLs no rule caught, compare slugs directly. Slugs survive migrations more often than paths do — the directory moves, the final segment barely changes. rapidfuzz handles this at scale (its cdist computes the full similarity matrix in C, so 4,000 × 15,000 comparisons take seconds, not hours):
from rapidfuzz import fuzz, process
import numpy as np
def slug(u):
return u.rstrip("/").split("/")[-1].replace("-", " ")
old_slugs = [slug(u) for u in unmatched_old]
new_slugs = [slug(u) for u in new_urls]
scores = process.cdist(old_slugs, new_slugs,
scorer=fuzz.token_sort_ratio, workers=-1)
best = scores.argmax(axis=1)
for i, j in enumerate(best):
top = scores[i, j] / 100
runner_up = np.partition(scores[i], -2)[-2] / 100
margin = top - runner_up
if top >= 0.90 and margin >= 0.15:
accept(unmatched_old[i], new_urls[j],
method="fuzzy", confidence=top)The margin check matters more than the absolute threshold. A 0.92 top score with a 0.91 runner-up means two new pages are near-identical candidates — exactly the row a human should see. A 0.90 score with nothing else above 0.60 is safe to auto-accept. Stage 2 resolved another 2,700 URLs (22%) for us.
Stage 3: content embeddings for the hard residue
What is left — typically 10–15% — is the residue where the URL tells you nothing: slugs were rewritten for keywords, pages were merged, templates changed. Here you match on what the page is about, using the same embedding approach we used for the internal linking suggestion pipeline: embed title + H1 + first ~2,000 characters of main content for every unmatched old page and every new page, then take cosine similarity.
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("all-MiniLM-L6-v2")
def doc(page):
return f"{page['title']}. {page['h1']}. {page['body'][:2000]}"
E_old = model.encode([doc(p) for p in residue], normalize_embeddings=True)
E_new = model.encode([doc(p) for p in new_pages], normalize_embeddings=True)
sim = E_old @ E_new.T
for i, row in enumerate(sim):
j = row.argmax()
if row[j] >= 0.80:
accept(residue[i]["url"], new_pages[j]["url"],
method="embedding", confidence=float(row[j]))
else:
review_queue.append((residue[i]["url"],
top_candidates(row, new_pages, k=3)))Embeddings correctly handle the cases that break lexical matching: a legacy page titled “Winter tyre regulations 2019” maps to “Seasonal tire law guide” at 0.86 similarity despite sharing almost no slug tokens. They also expose true deletions honestly — if the best cosine score across the entire new site is 0.45, that content genuinely no longer exists, and the correct answer is a 410, not a desperate redirect to something vaguely related.
Weight the review queue by traffic value, not row order
The review queue is where analyst hours go, so order it by what the URL is worth, not alphabetically. Blend GSC clicks, referring domains, and internal PageRank from your crawler:
df["traffic_value"] = (
0.5 * rank_pct(df.gsc_clicks_90d) +
0.3 * rank_pct(df.referring_domains) +
0.2 * rank_pct(df.internal_pagerank)
)
review = df[df.needs_review].sort_values("traffic_value", ascending=False)In practice the value distribution is brutally skewed: on our project, the top 300 URLs of the 1,800-row review queue held 91% of the queue’s total clicks. Review those carefully, then make an explicit policy decision for the tail — map to the closest section page where the embedding score is at least moderate, and 410 the rest. An honest 410 lets Google drop the URL quickly instead of grinding through soft-404 evaluations of a bad redirect.
Ship it as config, then verify against reality
The map should compile to your redirect layer automatically — nginx map blocks, Cloudflare Bulk Redirects via API, or an .htaccess file, all generated from the same CSV so the artifact under review is the artifact deployed. Two verification passes close the loop:
Pre-launch: replay every old_url against staging with the new redirect config and assert exactly one 301 hop landing on a 200. This catches chains (A→B where B is itself redirected) and loops before they ship. It also catches destinations that are noindexed or blocked — cross-check against your orphan page detection pipeline so you never redirect valuable equity into a page nothing links to.
Post-launch: your log files and GSC crawl stats are the ground truth. Watch 404s spike in the first 72 hours — every 404 with real traffic is a mapping you missed, and if you already run a 404 monitoring pipeline that suggests redirects, it becomes your safety net: it will surface the residue this pipeline could not resolve, now with live demand data attached.
What this changes in practice
On the 12,400-URL migration: 64% resolved by rules, 22% by fuzzy slugs, 8% by embeddings above threshold, 6% to human review — about 740 URLs, of which an analyst cleared the valuable 300 in one day. Organic clicks recovered to baseline in five weeks. The previous migration at the same company, mapped by hand, took nine months to recover and never fully did for the blog section, because 1,100 posts had been bulk-redirected to /blog/.
The pipeline did not remove humans from the process. It moved them from row 8,000 of a spreadsheet to the 300 decisions that actually needed judgment. That is the correct shape for almost every SEO automation problem.
FAQ
Should every old URL get a redirect?
No. URLs with no traffic, no backlinks, and no close content equivalent on the new site should return 410. Mass-redirecting everything to weakly related pages triggers soft-404 classification, which wastes crawl budget and transfers no equity anyway.
Which embedding model is good enough for URL matching?
A small local model like all-MiniLM-L6-v2 is sufficient because the task is relative ranking within one site, not open-domain retrieval. Larger API models raise scores uniformly but rarely change which candidate wins. Spend the effort on extracting clean main content instead.
How do I handle one-to-many splits, where one old page became three new pages?
A redirect can only point one place, so send the old URL to the new page with the highest embedding similarity — usually the most general of the three — and add on-page links to the siblings. Record the split in the map so the decision is auditable.
How long should migration redirects stay in place?
Treat them as permanent. Google has stated 301s need to persist for at least a year to fully transfer signals, but backlinks pointing at old URLs keep sending users indefinitely. Keep the map deployed; only prune entries that logs show received zero hits for 12+ months.
