Detect and Fix Keyword Cannibalization at Scale with Python and the GSC API

Keyword cannibalization is one of those problems that hides in plain sight. You publish two, three, sometimes five articles that all brush up against the same query, and instead of one page ranking strongly, Google shuffles between them — none of which ever lock into the top three. Clicks leak, impressions split, and your most authoritative URL gets demoted by its own siblings. Most teams “find” cannibalization by gut feel, usually after a ranking drop. This post shows how to detect it programmatically with the Google Search Console (GSC) API and a few dozen lines of Python, score each cluster by how much it is actually costing you, and decide what to consolidate, redirect, or leave alone.

What cannibalization actually looks like in the data

The textbook definition — “two URLs competing for the same keyword” — is too loose to act on. In GSC data, real cannibalization has a specific signature: a single query where more than one URL from your site receives impressions, and the ranking URL changes over time. That last part matters. If page A always ranks for the query and page B never surfaces, there is no conflict. The damage happens when Google can’t decide, so it rotates between candidates, resetting whatever ranking momentum each page was building.

So instead of flagging every query that two pages happen to share, we look for three signals together: multiple URLs with non-trivial impressions on the same query, an average position stuck outside the top positions, and instability in which URL is chosen. That combination is what separates a genuine cannibalization problem from harmless topical overlap.

Step 1 — Pull query-by-page data from the GSC API

The Search Console API lets you request the query and page dimensions together, which is exactly the join we need. The web UI hides this; the API gives it to you cleanly. Authenticate with a service account that has been added as a user in the GSC property, then request the last 90 days at the query+page grain.

from google.oauth2 import service_account
from googleapiclient.discovery import build
import pandas as pd

SITE = "https://example.com/"
creds = service_account.Credentials.from_service_account_file(
    "service_account.json",
    scopes=["https://www.googleapis.com/auth/webmasters.readonly"],
)
gsc = build("searchconsole", "v1", credentials=creds)

def fetch(start, end):
    rows, start_row = [], 0
    while True:
        resp = gsc.searchanalytics().query(siteUrl=SITE, body={
            "startDate": start, "endDate": end,
            "dimensions": ["query", "page"],
            "rowLimit": 25000, "startRow": start_row,
        }).execute()
        batch = resp.get("rows", [])
        rows.extend(batch)
        if len(batch) < 25000:
            break
        start_row += 25000
    return pd.DataFrame([{
        "query": r["keys"][0], "page": r["keys"][1],
        "clicks": r["clicks"], "impressions": r["impressions"],
        "position": r["position"],
    } for r in rows])

df = fetch("2026-03-31", "2026-06-29")

Pagination matters here. A mid-sized site easily blows past 25,000 rows at the query+page grain, and if you forget to page through, you silently analyze a truncated slice of your own data.

Step 2 — Isolate queries with competing URLs

Now group by query and keep only the ones where more than one URL earns a meaningful share of impressions. The impression floor is the single most important parameter in the whole script: set it too low and you drown in noise from long-tail queries that two pages incidentally share; set it too high and you miss real conflicts on mid-volume terms.

MIN_IMPRESSIONS = 50   # per URL, over the window
MIN_URLS = 2

competing = []
for query, g in df.groupby("query"):
    g = g[g["impressions"] >= MIN_IMPRESSIONS]
    if g["page"].nunique() < MIN_URLS:
        continue
    total_impr = g["impressions"].sum()
    best_pos = g["position"].min()
    competing.append({
        "query": query,
        "urls": g["page"].nunique(),
        "total_impressions": int(total_impr),
        "total_clicks": int(g["clicks"].sum()),
        "best_position": round(best_pos, 1),
        "pages": g.sort_values("impressions", ascending=False)["page"].tolist(),
    })

cannibal = pd.DataFrame(competing)

At this point you have a list of queries where your own URLs are genuinely sharing impressions. But not all of these are worth touching — which is what the scoring step is for.

Step 3 — Score each conflict by lost opportunity

Raw conflict counts are a vanity metric. What you want is a priority score that answers “if I fix this, how much traffic could I plausibly recover?” The logic: a conflict is expensive when the query has high impressions, the best position is mediocre (so there is upside), and the impressions are split fairly evenly across URLs (true competition rather than one dominant page with a straggler).

import numpy as np

def gini_split(impr_list):
    # 0 = one page dominates, 1 = perfectly even split
    arr = np.sort(np.array(impr_list, dtype=float))
    n = len(arr)
    if arr.sum() == 0:
        return 0.0
    gini = (2 * np.sum((np.arange(1, n + 1)) * arr) / (n * arr.sum())) - (n + 1) / n
    return round(1 - gini, 3)

def ctr_for_position(pos):
    # rough top-of-funnel CTR curve; calibrate to your own data
    table = {1: 0.28, 2: 0.15, 3: 0.10, 4: 0.07, 5: 0.05}
    return table.get(int(round(pos)), 0.02)

scores = []
for _, row in cannibal.iterrows():
    g = df[(df["query"] == row["query"]) & (df["impressions"] >= MIN_IMPRESSIONS)]
    evenness = gini_split(g["impressions"].tolist())
    potential_ctr = ctr_for_position(max(1, row["best_position"] - 2))
    current_clicks = row["total_clicks"]
    potential_clicks = row["total_impressions"] * potential_ctr
    upside = max(0, potential_clicks - current_clicks)
    scores.append(round(upside * evenness, 1))

cannibal["priority"] = scores
cannibal = cannibal.sort_values("priority", ascending=False)
cannibal.head(20).to_csv("cannibalization_priorities.csv", index=False)

The evenness term — a quick Gini-based measure of how impressions are distributed — is what stops a query where one page owns 95% of impressions from outranking a genuine 50/50 split in your priority list. The CTR uplift estimate is deliberately crude; the goal is a defensible ranking of where to spend your editorial hours, not a forecast you would put in a board deck.

Step 4 — Decide: consolidate, differentiate, or redirect

The script tells you where the problem is; the fix is an editorial judgment that automation should inform, not make for you. Three patterns cover almost every case. When two pages cover the same intent, consolidate: merge the weaker into the stronger, 301-redirect the loser, and fold its useful sections in. When the pages target genuinely different intents that Google is conflating, differentiate: rewrite titles, H1s, and opening paragraphs so each page sends an unambiguous relevance signal, and add a clear internal link from one to the other to tell Google which is canonical for which angle. When one page is simply obsolete, redirect it and move on.

This is where automation hands off to human review — and where it pays to wire the output into the rest of your stack. The same embedding approach from our guide to keyword clustering with embeddings and HDBSCAN can pre-group the conflicting URLs by semantic similarity so you can tell “same intent” from “different intent” at a glance, and the internal-link fixes slot neatly into a semantic internal linking engine rather than being applied by hand.

Step 5 — Run it on a schedule and watch the trend

Cannibalization is not a one-time cleanup; new conflicts appear every time you publish near an existing topic. Schedule the script weekly (cron, GitHub Actions, or an n8n cron node), append the top conflicts to a tracking sheet, and diff against last week’s run. A conflict that climbs your priority list two weeks running is a far stronger signal than any single snapshot. If you already maintain a reporting pipeline, this folds directly into it — see our walkthrough on building a self-updating SEO dashboard with GSC bulk export and BigQuery for a place to surface the trend.

Results you can expect

On the sites where we have run this, the pattern is consistent: the top five conflicts by priority score almost always involve a flagship page that has been quietly held at position 5–8 by a thinner article published months later. Consolidating those — redirect the straggler, merge its unique angles into the flagship — typically recovers the flagship to the top three within two to four weeks, because Google finally has one unambiguous candidate. The script does not do the recovery; it does the far harder thing of telling you, out of hundreds of overlapping queries, which five are actually worth your afternoon.

Takeaways

Treat cannibalization as a data problem first and an editorial problem second. Pull query+page data from the GSC API, flag only queries where multiple URLs share real impressions, score by recoverable upside rather than raw conflict count, and let a human make the consolidate-or-differentiate call on the short list. Run it weekly so new conflicts surface before they cost you a ranking, not after.

If you found this useful, bookmark SEOAutomationClub and check back for the weekly automation playbook — we ship a new working-code SEO workflow every morning.

Frequently asked questions

How is keyword cannibalization different from two pages just sharing a keyword?

Sharing a keyword is harmless if one page consistently ranks and the other never surfaces. True cannibalization shows a specific signature in GSC data: multiple URLs earn meaningful impressions on the same query AND the ranking URL changes over time, so Google never lets one page build momentum. Flag the instability, not the overlap.

Can I detect cannibalization from the Search Console UI instead of the API?

Not well. The UI lets you filter a query and see pages, but it won’t give you the full query+page matrix across thousands of queries at once. The API returns the query and page dimensions together so you can group, score, and rank conflicts programmatically — which is what makes this an automatable workflow rather than manual spot-checking.

What impression threshold should I use to flag a conflict?

Start at roughly 50 impressions per URL over a 90-day window and tune from there. Too low and long-tail queries that two pages incidentally share flood your list; too high and you miss real conflicts on mid-volume terms. The threshold is the single most influential parameter in the script, so adjust it to your site’s traffic scale.

Should I always 301-redirect the weaker page?

No. Redirect only when both pages serve the same search intent. If they target genuinely different intents that Google is conflating, differentiate them instead — rewrite titles, H1s and intros, and add a clear internal link establishing which page is canonical for which angle. Reserve redirects for pages that are truly redundant or obsolete.