Automate Keyword Clustering in Python: GSC + Embeddings + HDBSCAN Pipeline

Keyword research is where most SEO programs quietly leak time. A mid-sized site exports 4,000 queries from Search Console, and someone spends two days dragging rows around a spreadsheet, guessing which queries belong on the same page. The output is subjective, unrepeatable, and stale the moment traffic shifts. The fix isn’t a better spreadsheet — it’s treating clustering as a data pipeline you run on a schedule.

This post walks through an end-to-end keyword clustering pipeline you can run in Python: pull real queries from the Google Search Console (GSC) API, embed them with a sentence-transformer model, cluster them with HDBSCAN, and label each cluster’s search intent with a single LLM call. The result is a page-level content map that tells you which queries deserve one URL, which deserve their own, and where you already have coverage. Every code block below is runnable — SAC’s whole point is working code, not “10 reasons clustering matters.”

Why semantic clustering beats keyword-match grouping

The traditional approach groups keywords by shared tokens: everything containing “running shoes” goes in one bucket. That breaks in both directions. “Best running shoes for flat feet” and “running shoe arch support” share almost no tokens but are the same page. “Apple stock” and “apple pie recipe” share the token that matters least. Lexical grouping can’t see that, so you end up either splitting one topic across five thin pages (cannibalization) or cramming three intents onto one bloated URL.

Embeddings fix this by mapping each query to a vector where semantic neighbors sit close together, regardless of exact wording. Cluster those vectors and you get groups that mirror how a search engine — and increasingly an LLM answer engine — understands the topic. This is the same representation logic behind our keyword cannibalization detector; here we’re using it constructively to plan coverage instead of diagnosing overlap after the fact.

Step 1 — Pull real queries from the GSC API

Start from data you already rank for, not a scraped keyword dump. Search Console gives you queries with impressions and clicks, which lets you cluster and prioritize in the same pass. Authenticate with a service account that has access to your property:

from googleapiclient.discovery import build
from google.oauth2 import service_account

SITE = "https://example.com/"
creds = service_account.Credentials.from_service_account_file(
    "gsc-service-account.json",
    scopes=["https://www.googleapis.com/auth/webmasters.readonly"],
)
gsc = build("searchconsole", "v1", credentials=creds)

def fetch_queries(start, end, row_limit=25000):
    rows, start_row = [], 0
    while True:
        resp = gsc.searchanalytics().query(siteUrl=SITE, body={
            "startDate": start, "endDate": end,
            "dimensions": ["query"],
            "rowLimit": 25000, "startRow": start_row,
        }).execute()
        batch = resp.get("rows", [])
        rows.extend(batch)
        if len(batch) < 25000:
            break
        start_row += 25000
    return [
        {"query": r["keys"][0], "clicks": r["clicks"],
         "impressions": r["impressions"], "position": r["position"]}
        for r in rows
    ]

queries = fetch_queries("2026-04-01", "2026-06-30")
print(f"Pulled {len(queries)} queries")

Filter aggressively before clustering. Drop queries with fewer than ~10 impressions over the window — they add noise and cost embedding time without changing the map. Also strip near-duplicate whitespace and lowercase everything so the model isn’t distracted by casing.

Step 2 — Embed the queries locally

You don’t need a paid embedding API for this. A small open model like all-MiniLM-L6-v2 runs on CPU in seconds for a few thousand queries and produces 384-dimensional vectors that cluster well. Normalize the embeddings so cosine similarity behaves like Euclidean distance, which matters for the clustering step:

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")
texts = [q["query"] for q in queries]
emb = model.encode(texts, normalize_embeddings=True,
                   show_progress_bar=True, batch_size=64)
emb = np.asarray(emb, dtype="float32")

If you already run a semantic internal-linking job, you likely have this exact model cached — the pipeline behind our semantic internal linking engine uses the same encoder, so you can share the embedding cache across both jobs and save the compute.

Step 3 — Cluster with HDBSCAN, not K-Means

K-Means forces you to pick the number of clusters up front and assigns every point to a cluster, including the junk. For keyword data — where cluster sizes are wildly uneven and a chunk of long-tail queries genuinely belongs to no topic — HDBSCAN is the better tool. It finds clusters of varying density and, crucially, labels true outliers as -1 instead of shoving them into the nearest group.

import hdbscan

clusterer = hdbscan.HDBSCAN(
    min_cluster_size=4,        # smallest group that counts as a topic
    min_samples=2,             # lower = more clusters, higher = more noise
    metric="euclidean",        # valid on normalized vectors
    cluster_selection_method="eom",
)
labels = clusterer.fit_predict(emb)

for i, q in enumerate(queries):
    q["cluster"] = int(labels[i])

n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
noise = int((labels == -1).sum())
print(f"{n_clusters} clusters, {noise} unclustered queries")

Tune min_cluster_size to your site: 4 is a sensible start for a few thousand queries; raise it to 8–10 on very large exports so you don’t drown in micro-topics. The unclustered bucket isn’t waste — it’s your long-tail radar. Queries that refuse to cluster are often emerging topics or one-off informational intents worth a quick manual scan.

Step 4 — Rank clusters and pick a head term

Now attach business value. For each cluster, sum impressions, average position, and pick the highest-impression query as the working head term. This turns an abstract cluster into a prioritized content brief line:

from collections import defaultdict

groups = defaultdict(list)
for q in queries:
    if q["cluster"] != -1:
        groups[q["cluster"]].append(q)

summary = []
for cid, members in groups.items():
    members.sort(key=lambda x: x["impressions"], reverse=True)
    total_impr = sum(m["impressions"] for m in members)
    avg_pos = sum(m["position"] for m in members) / len(members)
    summary.append({
        "cluster": cid,
        "head_term": members[0]["query"],
        "size": len(members),
        "impressions": int(total_impr),
        "avg_position": round(avg_pos, 1),
        "samples": [m["query"] for m in members[:6]],
    })

summary.sort(key=lambda x: x["impressions"], reverse=True)

Sorting by impressions with an eye on average position gives you an instant priority list. A cluster with 40,000 impressions sitting at position 12 is a striking-distance opportunity — exactly the kind of near-miss our striking distance finder surfaces, now grouped at the topic level instead of query by query.

Step 5 — Label intent with one LLM call

The last manual step is classifying each cluster’s intent: informational, commercial, transactional, or navigational. Instead of eyeballing thousands of queries, send the head term plus a handful of samples to an LLM and get a structured label back. One batched call per run keeps cost negligible:

import json, os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def label_clusters(summary):
    payload = [{"cluster": s["cluster"], "head_term": s["head_term"],
                "samples": s["samples"]} for s in summary]
    prompt = (
        "For each keyword cluster, return JSON with fields cluster, "
        "intent (informational|commercial|transactional|navigational), "
        "and a 4-6 word suggested page title. Clusters:\n"
        + json.dumps(payload)
    )
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

intents = label_clusters(summary)

Merge the LLM labels back onto your summary and you have a finished content map: a ranked table of topics, each with a head term, an intent, a suggested title, member queries, and the traffic behind it. Export it to CSV, or push it straight into a Looker Studio dashboard so non-technical stakeholders can filter by intent and opportunity.

Results: what this changes in practice

On a 3,800-query export, this pipeline runs end to end in under two minutes on a laptop and produces roughly 180–220 clusters. The value isn’t just speed — it’s consistency. Because the same input always yields the same map, you can rerun it monthly and diff the output to see which topics are newly emerging, which clusters are drifting in position, and where a single URL has started absorbing two distinct intents (an early cannibalization warning).

Three decisions fall out of the map almost automatically. Clusters with one dominant high-impression query and tight semantic spread map to a single URL. Clusters that visibly contain two intent groups — the LLM often flags this by returning a low-confidence label — are candidates to split. And large unclustered residue in a topic area you care about signals a content gap worth a dedicated brief. That’s the whole point: move the judgment from “which of these 4,000 rows go together” to “which of these 200 topics do we act on.”

To wire this into a recurring system, drop the script into a scheduled job — cron, GitHub Actions, or an n8n workflow that fetches GSC, runs the clustering container, and writes the CSV to storage. If you want SERP context on top of the GSC data (who currently ranks for each head term), enrich the head terms with a scraping layer such as Bright Data before the labeling step. The architecture is the same one we used to automate content refresh briefs: pull, transform, enrich, decide.

Frequently asked questions

How many queries do I need before clustering is worth it?

Below about 300 queries you can group by hand faster than you can tune HDBSCAN. The pipeline earns its keep from roughly 1,000 queries up, and the value scales with export size — at 10,000+ queries manual grouping is effectively impossible to do consistently.

Why HDBSCAN instead of K-Means or agglomerative clustering?

Keyword data has uneven cluster sizes and a real noise population — long-tail queries that belong to no topic. HDBSCAN handles both: it finds variable-density clusters and explicitly labels outliers as -1 instead of forcing them into a group, which keeps your topics clean.

Do I need a paid embedding API?

No. A local sentence-transformer like all-MiniLM-L6-v2 runs on CPU and is more than accurate enough for query clustering. Paid embeddings can marginally improve separation on ambiguous terms, but for GSC query data the open model is the pragmatic default and keeps the whole pipeline free to run.

Can this replace a keyword research tool entirely?

It replaces the grouping and prioritization step, which is the slowest part. It doesn’t discover keywords you don’t already rank for — for net-new demand you still want a discovery source. The strongest setup uses this pipeline on your GSC data and layers external keyword discovery on top for expansion.

Take it further

If this kind of pull-transform-decide automation is how you want to run SEO, bookmark SAC and check back for weekly automation playbooks — every post ships with code you can paste and run. A natural next build is pairing this map with our cannibalization detector so the same embedding pass both plans new coverage and flags URLs already competing with each other.