| |

Search Intent Clustering in Python: GSC + Embeddings + HDBSCAN Pipeline

BySEO AutomationClub



Search intent clustering is the practice of grouping the queries a site already ranks for by the intent behind them — what the searcher is actually trying to do — rather than by the words they happen to share. Do it well and every cluster maps cleanly to one page and one job; do it badly and a single topic gets split across three thin URLs that cannibalize each other in the SERP.

This is the canonical, end-to-end pipeline for doing it in Python: pull real queries from the Search Console API, embed them locally, cluster them with HDBSCAN, then label each cluster’s intent with a single LLM call. It is built for SEOs and technical marketers handling a few hundred to a few thousand queries who want a repeatable process instead of a one-off tool export. If you have 50 keywords or never touch code, this is overkill — a spreadsheet will do. If you want the same logic wired into an automation that runs unattended, the companion build, keyword clustering at scale with n8n, picks up where this guide ends.

Keyword research is where most SEO programs quietly leak time. A mid-sized site exports 4,000 queries from Search Console, and someone spends two days dragging rows around a spreadsheet, guessing which queries belong on the same page. The output is subjective, unrepeatable, and stale the moment traffic shifts. The fix isn’t a better spreadsheet — it’s treating clustering as a data pipeline you run on a schedule.

This post walks through an end-to-end keyword clustering pipeline you can run in Python: pull real queries from the Google Search Console (GSC) API, embed them with a sentence-transformer model, cluster them with HDBSCAN, and label each cluster’s search intent with a single LLM call. The result is a page-level content map that tells you which queries deserve one URL, which deserve their own, and where you already have coverage. Every code block below is runnable — SAC’s whole point is working code, not “10 reasons clustering matters.”

Approach How queries get grouped Intent accuracy Repeatable? Best for
Manual spreadsheet grouping A human eyeballs word overlap Low — misses synonyms and paraphrases No Under 50 queries, one-off
Keyword-tool “grouping” Lexical or SERP-overlap rules Medium — SERP overlap is a proxy for intent, not intent itself Partly, but locked to the tool A quick first pass
Semantic search intent clustering (this pipeline) Sentence embeddings plus HDBSCAN density High — groups by meaning, then labels the intent explicitly Yes — a script you run on a schedule 200–5,000 queries, ongoing

Keyword clustering vs. search intent clustering: what actually differs

The two terms get used interchangeably, but they answer different questions. Plain keyword clustering asks “which of these strings are similar?” Search intent clustering adds a second question: “and what does each group of searchers want?” A cluster containing best rank tracker, rank tracker comparison, and rank tracker pricing is lexically tidy, but it mixes a commercial-investigation intent (comparison) with a transactional one (pricing). Merge them onto one page and you dilute both.

Embeddings help here because they encode meaning, not characters. How to track keyword rankings and keyword position monitoring share almost no words yet sit close in vector space, so density-based clustering pulls them together. The intent label — informational, commercial, navigational, or transactional — is what turns a tidy cluster into a page brief: it tells you whether the URL should be a tutorial, a comparison, a landing page, or a product. That is why Step 5 of this pipeline spends an LLM call on labelling intent rather than treating the cluster head term as the whole answer.

The practical payoff is fewer cannibalization mistakes. When intent is explicit, two clusters that look similar but serve different jobs stay on separate URLs, and two clusters that look different but serve the same job get merged before you ever write the brief.

Why semantic clustering beats keyword-match grouping

The traditional approach groups keywords by shared tokens: everything containing “running shoes” goes in one bucket. That breaks in both directions. “Best running shoes for flat feet” and “running shoe arch support” share almost no tokens but are the same page. “Apple stock” and “apple pie recipe” share the token that matters least. Lexical grouping can’t see that, so you end up either splitting one topic across five thin pages (cannibalization) or cramming three intents onto one bloated URL.

Embeddings fix this by mapping each query to a vector where semantic neighbors sit close together, regardless of exact wording. Cluster those vectors and you get groups that mirror how a search engine — and increasingly an LLM answer engine — understands the topic. This is the same representation logic behind our keyword cannibalization detector; here we’re using it constructively to plan coverage instead of diagnosing overlap after the fact.

Step 1 — Pull real queries from the GSC API

Start from data you already rank for, not a scraped keyword dump. Search Console gives you queries with impressions and clicks, which lets you cluster and prioritize in the same pass. Authenticate with a service account that has access to your property:

from googleapiclient.discovery import build
from google.oauth2 import service_account

SITE = "https://example.com/"
creds = service_account.Credentials.from_service_account_file(
    "gsc-service-account.json",
    scopes=["https://www.googleapis.com/auth/webmasters.readonly"],
)
gsc = build("searchconsole", "v1", credentials=creds)

def fetch_queries(start, end, row_limit=25000):
    rows, start_row = [], 0
    while True:
        resp = gsc.searchanalytics().query(siteUrl=SITE, body={
            "startDate": start, "endDate": end,
            "dimensions": ["query"],
            "rowLimit": 25000, "startRow": start_row,
        }).execute()
        batch = resp.get("rows", [])
        rows.extend(batch)
        if len(batch) < 25000:
            break
        start_row += 25000
    return [
        {"query": r["keys"][0], "clicks": r["clicks"],
         "impressions": r["impressions"], "position": r["position"]}
        for r in rows
    ]

queries = fetch_queries("2026-04-01", "2026-06-30")
print(f"Pulled {len(queries)} queries")

Filter aggressively before clustering. Drop queries with fewer than ~10 impressions over the window — they add noise and cost embedding time without changing the map. Also strip near-duplicate whitespace and lowercase everything so the model isn’t distracted by casing.

Step 2 — Embed the queries locally

You don’t need a paid embedding API for this. A small open model like all-MiniLM-L6-v2 runs on CPU in seconds for a few thousand queries and produces 384-dimensional vectors that cluster well. Normalize the embeddings so cosine similarity behaves like Euclidean distance, which matters for the clustering step:

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")
texts = [q["query"] for q in queries]
emb = model.encode(texts, normalize_embeddings=True,
                   show_progress_bar=True, batch_size=64)
emb = np.asarray(emb, dtype="float32")

If you already run a semantic internal-linking job, you likely have this exact model cached — the pipeline behind our semantic internal linking engine uses the same encoder, so you can share the embedding cache across both jobs and save the compute.

Step 3 — Cluster with HDBSCAN, not K-Means

K-Means forces you to pick the number of clusters up front and assigns every point to a cluster, including the junk. For keyword data — where cluster sizes are wildly uneven and a chunk of long-tail queries genuinely belongs to no topic — HDBSCAN is the better tool. It finds clusters of varying density and, crucially, labels true outliers as -1 instead of shoving them into the nearest group.

import hdbscan

clusterer = hdbscan.HDBSCAN(
    min_cluster_size=4,        # smallest group that counts as a topic
    min_samples=2,             # lower = more clusters, higher = more noise
    metric="euclidean",        # valid on normalized vectors
    cluster_selection_method="eom",
)
labels = clusterer.fit_predict(emb)

for i, q in enumerate(queries):
    q["cluster"] = int(labels[i])

n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
noise = int((labels == -1).sum())
print(f"{n_clusters} clusters, {noise} unclustered queries")

Tune min_cluster_size to your site: 4 is a sensible start for a few thousand queries; raise it to 8–10 on very large exports so you don’t drown in micro-topics. The unclustered bucket isn’t waste — it’s your long-tail radar. Queries that refuse to cluster are often emerging topics or one-off informational intents worth a quick manual scan.

Step 4 — Rank clusters and pick a head term

Now attach business value. For each cluster, sum impressions, average position, and pick the highest-impression query as the working head term. This turns an abstract cluster into a prioritized content brief line:

from collections import defaultdict

groups = defaultdict(list)
for q in queries:
    if q["cluster"] != -1:
        groups[q["cluster"]].append(q)

summary = []
for cid, members in groups.items():
    members.sort(key=lambda x: x["impressions"], reverse=True)
    total_impr = sum(m["impressions"] for m in members)
    avg_pos = sum(m["position"] for m in members) / len(members)
    summary.append({
        "cluster": cid,
        "head_term": members[0]["query"],
        "size": len(members),
        "impressions": int(total_impr),
        "avg_position": round(avg_pos, 1),
        "samples": [m["query"] for m in members[:6]],
    })

summary.sort(key=lambda x: x["impressions"], reverse=True)

Sorting by impressions with an eye on average position gives you an instant priority list. A cluster with 40,000 impressions sitting at position 12 is a striking-distance opportunity — exactly the kind of near-miss our striking distance finder surfaces, now grouped at the topic level instead of query by query.

Step 5 — Label intent with one LLM call

The last manual step is classifying each cluster’s intent: informational, commercial, transactional, or navigational. Instead of eyeballing thousands of queries, send the head term plus a handful of samples to an LLM and get a structured label back. One batched call per run keeps cost negligible:

import json, os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def label_clusters(summary):
    payload = [{"cluster": s["cluster"], "head_term": s["head_term"],
                "samples": s["samples"]} for s in summary]
    prompt = (
        "For each keyword cluster, return JSON with fields cluster, "
        "intent (informational|commercial|transactional|navigational), "
        "and a 4-6 word suggested page title. Clusters:\n"
        + json.dumps(payload)
    )
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

intents = label_clusters(summary)

Merge the LLM labels back onto your summary and you have a finished content map: a ranked table of topics, each with a head term, an intent, a suggested title, member queries, and the traffic behind it. Export it to CSV, or push it straight into a Looker Studio dashboard so non-technical stakeholders can filter by intent and opportunity.

Results: what this changes in practice

On a 3,800-query export, this pipeline runs end to end in under two minutes on a laptop and produces roughly 180–220 clusters. The value isn’t just speed — it’s consistency. Because the same input always yields the same map, you can rerun it monthly and diff the output to see which topics are newly emerging, which clusters are drifting in position, and where a single URL has started absorbing two distinct intents (an early cannibalization warning).

Three decisions fall out of the map almost automatically. Clusters with one dominant high-impression query and tight semantic spread map to a single URL. Clusters that visibly contain two intent groups — the LLM often flags this by returning a low-confidence label — are candidates to split. And large unclustered residue in a topic area you care about signals a content gap worth a dedicated brief. That’s the whole point: move the judgment from “which of these 4,000 rows go together” to “which of these 200 topics do we act on.”

To wire this into a recurring system, drop the script into a scheduled job — cron, GitHub Actions, or an n8n workflow that fetches GSC, runs the clustering container, and writes the CSV to storage. If you want SERP context on top of the GSC data (who currently ranks for each head term), enrich the head terms with a scraping layer such as Bright Data before the labeling step. The architecture is the same one we used to automate content refresh briefs: pull, transform, enrich, decide.

Frequently asked questions

How many queries do I need before clustering is worth it?

Below about 300 queries you can group by hand faster than you can tune HDBSCAN. The pipeline earns its keep from roughly 1,000 queries up, and the value scales with export size — at 10,000+ queries manual grouping is effectively impossible to do consistently.

Why HDBSCAN instead of K-Means or agglomerative clustering?

Keyword data has uneven cluster sizes and a real noise population — long-tail queries that belong to no topic. HDBSCAN handles both: it finds variable-density clusters and explicitly labels outliers as -1 instead of forcing them into a group, which keeps your topics clean.

Do I need a paid embedding API?

No. A local sentence-transformer like all-MiniLM-L6-v2 runs on CPU and is more than accurate enough for query clustering. Paid embeddings can marginally improve separation on ambiguous terms, but for GSC query data the open model is the pragmatic default and keeps the whole pipeline free to run.

Can this replace a keyword research tool entirely?

It replaces the grouping and prioritization step, which is the slowest part. It doesn’t discover keywords you don’t already rank for — for net-new demand you still want a discovery source. The strongest setup uses this pipeline on your GSC data and layers external keyword discovery on top for expansion.

The verdict: when this pipeline is worth it

If you manage a few hundred to a few thousand queries and revisit your content map more than once a quarter, treating search intent clustering as a scheduled script pays for itself quickly — the grouping is repeatable, defensible, and updates itself as traffic shifts. If your keyword set is tiny or static, a spreadsheet is genuinely faster and this is over-engineering. For teams in between who want the clustering to run unattended on new exports, move on to the n8n automation of this same pipeline; this guide is the reference for the logic, that one is the reference for running it at scale.

Take it further

If this kind of pull-transform-decide automation is how you want to run SEO, bookmark SAC and check back for weekly automation playbooks — every post ships with code you can paste and run. A natural next build is pairing this map with our cannibalization detector so the same embedding pass both plans new coverage and flags URLs already competing with each other.

Similar Posts