Vector Databases for SEO at Scale: pgvector vs Qdrant vs Pinecone vs Weaviate

If you have been following along with the embedding-powered workflows we keep publishing here — keyword clustering with HDBSCAN, content cannibalization detection, the semantic internal linking engine, and 301 redirect mapping — you have probably hit the same wall we did. Generating embeddings is the easy part. Storing them so you can query a quarter-million URLs in milliseconds is where the project quietly stalls.

Most SEO tutorials hand-wave this step with a NumPy array and a cosine_similarity loop. That works until your corpus crosses roughly 10,000 vectors, at which point an O(n²) similarity matrix turns a two-minute job into a coffee break, then into an out-of-memory crash. This post is the storage layer those tutorials skip: a head-to-head on the four vector backends we have actually run in production SEO pipelines — pgvector, Qdrant, Pinecone, and Weaviate — with the numbers, the trade-offs, and the working code to wire each one in.

Why a flat NumPy array stops scaling

A brute-force nearest-neighbor search compares your query vector against every stored vector. For a 5,000-page site that is fine. For a programmatic-SEO property with 250,000 URLs and 1,024-dimension embeddings, you are holding roughly a gigabyte of floats in RAM and running 250,000 dot products per query. Internal linking alone might issue one query per page, so you are looking at 62.5 billion operations for a single full pass.

Vector databases solve this with an approximate nearest neighbor (ANN) index — usually HNSW (Hierarchical Navigable Small World graphs) — that trades a sliver of recall for a 100-1000x speedup. Instead of scanning everything, HNSW walks a layered graph and converges on the closest neighbors in logarithmic time. For SEO work, where “the 20 most semantically related URLs” does not need to be mathematically perfect, that trade is almost always worth it.

The four contenders, by SEO use case

We are not benchmarking these for a RAG chatbot. The workloads that matter for SEO are bulk upserts after a crawl, periodic full-corpus similarity passes (cannibalization, clustering), and occasional point queries (find related pages for a new article). Here is how each backend lands against that reality.

pgvector — the default you probably already have

pgvector is a Postgres extension. If you already run Postgres — and most teams storing crawl data do — this is the lowest-friction option in existence. Your embeddings live in the same database as your URL metadata, status codes, and GSC clicks, so a “find cannibalizing pages that both rank on page 2” query is a single SQL join. No second system to sync, no network hop, no extra bill.

-- one-time setup
CREATE EXTENSION IF NOT EXISTS vector;
ALTER TABLE pages ADD COLUMN embedding vector(1024);

-- HNSW index (pgvector 0.5+) for fast cosine search
CREATE INDEX ON pages USING hnsw (embedding vector_cosine_ops);

-- find the 20 most similar URLs to a given page
SELECT url, 1 - (embedding <=> (SELECT embedding FROM pages WHERE id = 42)) AS similarity
FROM pages
WHERE id != 42
ORDER BY embedding <=> (SELECT embedding FROM pages WHERE id = 42)
LIMIT 20;

The catch: pgvector’s HNSW index build is memory-hungry and single-threaded on older versions, and it is happiest below a few million vectors. For most SEO properties that ceiling is irrelevant — you will run out of content before you run out of pgvector.

Qdrant — the purpose-built open-source pick

Qdrant is a dedicated vector database written in Rust. It is the one we reach for when the corpus is large or when filtering matters — and in SEO, filtering almost always matters. You rarely want “the most similar page”; you want “the most similar page that is indexable, returns 200, and is in the /blog/ subfolder.” Qdrant’s payload filtering is first-class and applied during the ANN search rather than after it, so you do not blow your recall by filtering a tiny result set post-hoc.

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct, Filter, FieldCondition, MatchValue

client = QdrantClient(url="http://localhost:6333")
client.recreate_collection(
    collection_name="pages",
    vectors_config=VectorParams(size=1024, distance=Distance.COSINE),
)

# bulk upsert after a crawl
client.upsert(collection_name="pages", points=[
    PointStruct(id=p["id"], vector=p["embedding"],
                payload={"url": p["url"], "indexable": p["indexable"], "folder": p["folder"]})
    for p in crawl_results
])

# similar pages, filtered to indexable /blog/ URLs only
hits = client.search(
    collection_name="pages",
    query_vector=target_embedding,
    query_filter=Filter(must=[
        FieldCondition(key="indexable", match=MatchValue(value=True)),
        FieldCondition(key="folder", match=MatchValue(value="/blog/")),
    ]),
    limit=20,
)

Qdrant runs as a single Docker container, scales to tens of millions of vectors on one box, and has a generous free Cloud tier if you do not want to self-host. For our money it is the best all-rounder for serious SEO automation.

Pinecone — fully managed, zero ops

Pinecone is the “I never want to think about infrastructure” choice. It is fully managed, serverless, and genuinely fast to stand up — you can be upserting vectors in ten minutes. The trade-off is that it is a closed SaaS with usage-based pricing that can climb as your corpus and query volume grow, and your embeddings now live in someone else’s cloud, decoupled from your crawl database. For an agency running many client sites without a platform team, that convenience can be worth paying for. For a single in-house property, the bill is harder to justify against Qdrant or pgvector.

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key="YOUR_KEY")
pc.create_index(name="pages", dimension=1024, metric="cosine",
                spec=ServerlessSpec(cloud="aws", region="us-east-1"))
index = pc.Index("pages")

index.upsert(vectors=[(str(p["id"]), p["embedding"], {"url": p["url"]}) for p in crawl_results])
res = index.query(vector=target_embedding, top_k=20, include_metadata=True,
                  filter={"indexable": True})

Weaviate — when you want hybrid search baked in

Weaviate’s differentiator is native hybrid search: it fuses dense vector similarity with BM25 keyword scoring in a single query. For SEO that is interesting because pure semantic search sometimes misses exact-match terms that still matter for intent — a query for “robots.txt” should not drift toward “crawl directives” just because they are semantically close. Weaviate also has built-in vectorizer modules that can call your embedding provider for you. The cost is a heavier, more opinionated system to operate than Qdrant, with a steeper schema learning curve.

Head-to-head comparison

Factor	pgvector	Qdrant	Pinecone	Weaviate
Model	Postgres extension	Open-source / Cloud	Managed SaaS	Open-source / Cloud
Best for	You already run Postgres	Large corpus + filtering	Zero-ops agencies	Hybrid (vector + keyword)
Practical scale	< ~2M vectors	Tens of millions	Effectively unlimited	Tens of millions
Metadata filtering	SQL WHERE (excellent)	First-class, pre-filter	Good	Good
Ops burden	None (existing DB)	Low (one container)	None	Medium
Joins with crawl data	Native	Via payload	External	Via properties
Cost shape	Your DB instance	Free self-host	Usage-based	Free self-host

What we actually run, and why

For sites under a couple million URLs that already store crawl output in Postgres — which is the majority of properties — we default to pgvector. Keeping embeddings beside the metadata removes an entire class of sync bugs, and a vector search becomes just another column in the same query that already filters by status code and indexability. The day a pipeline outgrows it — usually a large programmatic property or a multi-client crawl warehouse — we migrate the vector workload to Qdrant for the headroom and the pre-filtered ANN search, while keeping metadata in Postgres and joining on the URL key.

Pinecone earns its place when there is no platform team and ops time is the scarcest resource; the premium buys you never paging anyone at 2 a.m. Weaviate is the specialist pick when hybrid keyword-plus-semantic ranking is core to the workflow rather than a nice-to-have. The mistake we see most often is reaching for a managed SaaS on day one for a 30,000-URL site that pgvector would handle on a $15 database instance.

One more practical note: your choice of embedding model dictates your vector dimension, and dimension drives both index size and query speed. A 1,536-dimension OpenAI vector costs noticeably more to store and search than a 1,024-dimension Voyage or BGE vector across hundreds of thousands of rows. Pick the model and the database together, not in sequence.

Migration without re-embedding everything

Re-embedding a large corpus is expensive, so when you outgrow one backend, export the vectors you already have rather than regenerating them. Every tool above accepts raw float arrays on upsert, so the migration is a read-from-A, write-to-B loop — not a re-run of your embedding bill.

# pgvector -> Qdrant, reusing existing vectors
import psycopg2, ast
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct

pg = psycopg2.connect("dbname=seo")
cur = pg.cursor()
cur.execute("SELECT id, url, embedding FROM pages WHERE embedding IS NOT NULL")

qc = QdrantClient(url="http://localhost:6333")
batch = []
for pid, url, emb in cur:
    vec = ast.literal_eval(emb) if isinstance(emb, str) else list(emb)
    batch.append(PointStruct(id=pid, vector=vec, payload={"url": url}))
    if len(batch) == 1000:
        qc.upsert(collection_name="pages", points=batch); batch = []
if batch:
    qc.upsert(collection_name="pages", points=batch)

Key takeaways

The vector database is the unglamorous foundation that decides whether your embedding-based SEO automation scales or collapses. Start with pgvector if you already run Postgres — it is almost certainly enough, and it keeps your vectors next to your crawl data. Graduate to Qdrant when corpus size or filtering complexity demands it. Reach for Pinecone to buy away ops, and Weaviate when hybrid search is a first-class requirement. Above all, pick the backend and the embedding model as a pair, and never re-embed when a plain export will do.

Found this useful? Bookmark SEO Automation Club and check back — we publish working automation playbooks like this one every week. If you are wiring embeddings into your stack, start with our breakdown of which embedding model to use for SEO, then put your new vector store to work with the semantic internal linking engine.

Frequently asked questions

Do I really need a vector database for a small site?

No. Below roughly 10,000 URLs, an in-memory NumPy array with scikit-learn’s NearestNeighbors is perfectly fast and has zero infrastructure. The case for a vector database starts when full-corpus similarity passes get slow, when you cannot fit all vectors in RAM, or when you need filtered queries against live metadata.

HNSW or IVF — which index should I use?

For almost all SEO workloads, HNSW. It gives the best recall-versus-speed trade-off and does not require a training step. IVF-based indexes can use less memory at very large scale but need tuning and a representative training set, which is rarely worth the effort for SEO corpus sizes.

Will an approximate index hurt my SEO results?

In practice, no. ANN indexes are tunable, and at default settings they return well over 95% of the true nearest neighbors. For tasks like “find the 20 most related pages to link from,” missing one borderline candidate has no measurable SEO impact, while the speed gain makes full-corpus automation feasible at all.

Can I keep my crawl metadata in Postgres but vectors elsewhere?

Yes, and it is a common pattern at scale. Store the URL as a shared key, keep status codes, GSC data, and crawl attributes in Postgres, and keep vectors in Qdrant or Pinecone. Query the vector store for nearest neighbors, then join the returned IDs back to Postgres for the rich metadata.