Embedding Models for SEO, Compared: OpenAI vs Voyage vs Cohere vs Open-Source BGE

If you have read almost any modern SEO automation workflow on this site, you have quietly relied on text embeddings. Cannibalization detection compares page vectors. Redirect mapping at migration time matches old URLs to new ones by semantic similarity. Content gap analysis clusters query intent. Every one of those pipelines has a single hidden dependency: the embedding model you chose. Get that choice wrong and your similarity thresholds drift, your clusters smear together, and your monthly API bill quietly balloons.

The problem is that “which embedding model is best” is the wrong question for SEO work. The leaderboards everyone cites — MTEB and its descendants — are optimized for general retrieval and question answering. SEO embedding tasks are narrower and weirder: short, keyword-dense titles; near-duplicate templated pages; thin product descriptions; and similarity decisions where the cost of a false positive (merging two pages that should rank separately) is very different from a false negative. This post is a practitioner’s framework for picking an embedding model for SEO specifically, with the cost math, the dimension trade-off, and a reproducible benchmark you can run on your own corpus before you commit.

The four-way landscape in mid-2026

As of mid-2026 the practical shortlist for SEO pipelines comes down to four families, and they sort cleanly by where they sit on the cost-versus-control axis.

OpenAI text-embedding-3 (small and large)

Still the default for most teams because it is the path of least resistance: one API key, generous batch sizes (up to 2,048 inputs per request), and Matryoshka representation so you can truncate text-embedding-3-large from 3,072 dimensions down to 256 with graceful quality loss. The small variant sits at roughly $0.02 per million tokens, which makes it the cheapest commercial option for high-volume crawl work. The catch: it is a generalist, and on highly templated programmatic pages it tends to over-cluster because boilerplate dominates the signal.

Voyage AI (voyage-3-large and lite variants)

Voyage has become the quality benchmark for retrieval at production scale. For SEO the relevant feature is that it asks you to declare whether each input is a document or a query and applies different processing to each — which maps perfectly onto rank-tracking and intent-clustering work, where “the query a user typed” and “the page that should answer it” are genuinely different objects. It is more expensive than OpenAI small and caps batches at 128 inputs, so your throughput code has to chunk more aggressively.

Cohere embed-v3 (plus rerank)

Cohere’s distinctive move is pairing a decent embedding model with a strong reranker. For SEO that two-stage pattern is underrated: embed cheaply to retrieve a candidate set, then rerank the top 50 with the cross-encoder before you make a merge-or-keep decision. English-light sits around $0.02 per million tokens and the full English/multilingual models around $0.10. Like Voyage it distinguishes document and query inputs, and it caps batches at 96.

Open-source: BGE-M3 and Jina v5

If you are embedding millions of URLs a month, the API meter is the enemy, and self-hosting wins. BGE-M3 is the workhorse free, multilingual choice and runs comfortably on a single GPU; Jina’s v5 line pushes long context (32K tokens) and very wide language coverage. The trade is operational: you own the inference box, the batching, and the version pinning. For a migration you run twice a year, that overhead is not worth it. For a daily pipeline over a large site, it pays for itself in weeks.

The dimension trap nobody warns you about

The single most common mistake in SEO embedding pipelines is treating dimensions as “more is better.” Higher-dimensional vectors cost more to store, more to index in your vector database, and more compute on every similarity calculation — and for SEO similarity decisions the marginal accuracy above ~512 dimensions is usually noise.

Matryoshka models make this a tunable lever rather than a fixed cost. With text-embedding-3-large you can request 3,072 dims, then truncate and renormalize to 512 or 256 client-side. The right way to choose is empirical: embed a labeled sample, sweep the dimension count, and watch where your task metric stops improving.

import numpy as np
from openai import OpenAI

client = OpenAI()

def embed(texts, dims):
    # text-embedding-3 supports native truncation via the dimensions param
    resp = client.embeddings.create(
        model="text-embedding-3-large",
        input=texts,
        dimensions=dims,
    )
    return np.array([d.embedding for d in resp.data])

def cosine(a, b):
    a = a / np.linalg.norm(a, axis=1, keepdims=True)
    b = b / np.linalg.norm(b, axis=1, keepdims=True)
    return a @ b.T

# Two pages you KNOW should be treated as distinct, and two that cannibalize
labeled = {
    "distinct":   ("Best running shoes for flat feet", "How to clean white sneakers"),
    "duplicate":  ("Best running shoes for flat feet", "Top running shoes for fallen arches"),
}

for dims in (256, 512, 1024, 3072):
    for label, (a, b) in labeled.items():
        v = embed([a, b], dims)
        sim = float(cosine(v[:1], v[1:]))
        print(f"dims={dims:>4}  {label:<9}  cos={sim:.3f}")

Run that on your own labeled pairs and you will usually find that the gap between your "distinct" and "duplicate" similarity scores — the thing your threshold actually depends on — is nearly as wide at 512 dimensions as at 3,072, at a fraction of the storage cost. That gap, not the absolute score, is what makes a cannibalization detector reliable.

A reproducible benchmark for your own corpus

Leaderboards do not know your site. The only honest way to choose is to score the candidate models on a small hand-labeled set drawn from your actual pages. Build a gold set of 30–50 URL pairs labeled "should merge" or "keep separate" — pull the candidates straight from a crawl — then measure how cleanly each model separates the two classes.

from sklearn.metrics import roc_auc_score

# pairs: list of (text_a, text_b, label)  label=1 means "should merge"
def score_model(embed_fn, pairs, dims=512):
    sims, labels = [], []
    for a, b, y in pairs:
        v = embed_fn([a, b], dims)
        v = v / np.linalg.norm(v, axis=1, keepdims=True)
        sims.append(float(v[0] @ v[1]))
        labels.append(y)
    # AUC tells you how separable merge vs keep-separate is,
    # independent of where you eventually set the threshold
    return roc_auc_score(labels, sims)

# auc_openai  = score_model(embed_openai,  pairs)
# auc_voyage  = score_model(embed_voyage,  pairs)
# auc_bge     = score_model(embed_bge,     pairs)

AUC is the right metric here precisely because it is threshold-free: it tells you which model gives you the cleanest separation between pages that cannibalize and pages that do not, before you ever pick the cutoff. A model that scores 0.94 AUC on your corpus and costs $0.02/M is a better SEO choice than one that scores 0.95 at $0.18/M. Cheap-and-separable beats expensive-and-marginally-better, because the threshold absorbs the rest.

Cost math at SEO scale

The numbers only get scary when you multiply. Suppose a 200,000-URL site, re-embedded weekly to catch content changes, averaging ~400 tokens of title-plus-meta-plus-first-paragraph per page. That is roughly 80M tokens per run, ~4.2B tokens a year.

At $0.02 per million tokens that is about $84 a year — trivial, run it on OpenAI small and never think about it again. At $0.18 per million it is around $750 a year for the same job, and now self-hosting BGE-M3 on a spot GPU starts to look attractive. The decision rule writes itself: below a few hundred thousand pages, use the cheapest commercial model that clears your AUC bar; above that, benchmark a self-hosted open-source model against the same bar and compare total cost of ownership, not sticker price. If you are already running scraping infrastructure such as Bright Data or a similar backbone, you likely already have the orchestration to host inference too.

Practical recommendations by task

For cannibalization detection on a small-to-mid site, OpenAI text-embedding-3-small at 512 dimensions is the boring correct answer. For redirect mapping during a migration — a one-off, high-stakes job — spend up: Voyage or Cohere with document/query distinction, plus a reranking pass on ambiguous matches, because the cost of a wrong redirect is real. For intent clustering and content gap work, the document/query split in Voyage or Cohere genuinely helps. And for any daily pipeline over a large site, benchmark BGE-M3 first; the operational cost is usually worth the metered savings.

Key takeaways

Pick the model on your own corpus, not a leaderboard. Use AUC on a hand-labeled merge/keep-separate set to compare candidates threshold-free. Treat dimensions as a tunable cost lever and stop around 512 for most SEO tasks. Match the model to the stakes: cheap generalist for routine monitoring, premium plus reranking for irreversible decisions like redirects, self-hosted open-source once volume makes the API meter the bottleneck.

Found this useful? Bookmark SEOAutomationClub and check back — we publish working SEO automation playbooks with real code every week. If you are building similarity pipelines, start with our deep-dive on detecting content cannibalization with vector embeddings, then wire the model you chose here straight into it.

Frequently asked questions

Do I need a vector database to run these SEO embedding workflows?

Not for small jobs. Up to a few tens of thousands of vectors you can hold embeddings in memory as a NumPy array and compute cosine similarity directly, as in the snippets above. A dedicated vector store (pgvector, Qdrant, Pinecone) only earns its keep when you need persistent indexes, incremental updates, and sub-second nearest-neighbor search over hundreds of thousands of pages.

How often should I re-embed my pages?

Re-embed when content changes, not on a fixed clock. The efficient pattern is to hash each page's relevant text and only re-embed URLs whose hash changed since the last run. That turns a weekly full re-embed into a small incremental job and cuts your token bill dramatically on stable sites.

Can I mix embedding models across pipelines?

Yes, but never compare vectors produced by different models — their spaces are not aligned, so cosine similarity between an OpenAI vector and a BGE vector is meaningless. It is perfectly fine to use a cheap model for daily monitoring and a premium one for a migration, as long as each similarity comparison stays within a single model's space.

Why use AUC instead of just picking a similarity threshold?

The threshold is site- and model-specific, so comparing models at a fixed threshold is unfair. AUC measures how well a model separates the two classes across every possible threshold, which lets you compare candidates first and tune the cutoff second — the order that actually matches how you deploy.