Automate Keyword Cannibalization Detection: A Python Pipeline That Uses Search Console to Find Pages Fighting Each Other

Keyword cannibalization is one of the most misdiagnosed problems in technical SEO. Most audits flag it by eyeballing title tags or running a spreadsheet of pages that look similar. But similarity is not cannibalization. Two pages can share a keyword in the title and never compete, while two pages with completely different copy can quietly split clicks for the exact same query. The only source that knows the truth is your Search Console query data, because it records which URLs Google actually surfaced for each search.

This guide builds a Python pipeline that detects genuine cannibalization from GSC data: cases where two or more URLs on your site rotate in and out of the results for the same query, dragging each other down. It is a data problem, not a content-similarity problem, and treating it that way is what separates a useful audit from a pile of false positives.

Why title-based cannibalization checks fail

The classic approach exports a crawl, groups pages by target keyword, and flags duplicates. It produces two kinds of errors at once. It raises false alarms on pages that mention a keyword but rank for entirely different intents, and it misses the dangerous cases where two URLs compete for a query neither of them was written to target. This is the same trap that catches naive near-duplicate detection built on content similarity: what a page is about is a weak predictor of what it ranks for.

Real cannibalization has a signature in the data. For a single query, impressions are split across multiple URLs, average position is worse than either URL achieves alone on its good days, and the winning URL flips from week to week. None of that is visible in a crawl. All of it is visible in Search Console.

The data you actually need

Pull the Search Console Performance report at the query and page dimension together. The Search Analytics API returns rows keyed by the combination of query and page, which is exactly the grain cannibalization lives at. Request a rolling window of at least 28 days so a single volatile week does not distort the picture, and keep the raw rows rather than any pre-aggregated view.

Requesting query and page rows

from googleapiclient.discovery import build

def fetch_rows(service, site_url, start, end):
    rows, start_row = [], 0
    while True:
        resp = service.searchanalytics().query(
            siteUrl=site_url,
            body={
                "startDate": start,
                "endDate": end,
                "dimensions": ["query", "page"],
                "rowLimit": 25000,
                "startRow": start_row,
            },
        ).execute()
        batch = resp.get("rows", [])
        rows.extend(batch)
        if len(batch) < 25000:
            break
        start_row += 25000
    return rows

Each row gives you the query, the page, and the clicks, impressions, CTR, and position for that pair. Load it into a pandas DataFrame and split the two keys into their own columns so you can group by query.

Defining cannibalization in measurable terms

The pipeline needs a rule that a machine can apply consistently. A query is cannibalized when three conditions hold together. First, two or more URLs each earn a meaningful share of impressions for that query, not a token handful. Second, both URLs rank close enough that they plausibly swap positions rather than one sitting on page one and the other on page five. Third, the query has enough total volume that the split actually costs you traffic.

Scoring each query

import pandas as pd

def cannibal_report(df, min_impr=100, pos_gap=8, min_share=0.15):
    df = df.copy()
    df[["query", "page"]] = pd.DataFrame(df["keys"].tolist(), index=df.index)
    out = []
    for q, grp in df.groupby("query"):
        total = grp["impressions"].sum()
        if total < min_impr:
            continue
        grp = grp.sort_values("impressions", ascending=False)
        top = grp.head(2)
        if len(top) < 2:
            continue
        shares = top["impressions"] / total
        if shares.min() < min_share:
            continue
        if (top["position"].max() - top["position"].min()) > pos_gap:
            continue
        out.append({
            "query": q,
            "urls": top["page"].tolist(),
            "impressions": int(total),
            "min_share": round(float(shares.min()), 2),
            "best_pos": round(float(top["position"].min()), 1),
        })
    return pd.DataFrame(out).sort_values("impressions", ascending=False)

The thresholds are levers, not laws. On a large site raise min_impr so you only surface queries worth the engineering time. On a small site lower it, but expect noisier results. The pos_gap guard is what keeps this honest: if one URL sits at position 3 and another at position 40, they are not fighting, and including them would flood your report with the same false positives that make title-based audits useless.

Confirming the flip with a time dimension

Impression share tells you two URLs share a query today. It does not tell you they are unstable. The strongest evidence of harmful cannibalization is positional volatility: the URL Google prefers keeps changing. Pull the same query and page data broken out by week, then measure how often the top-ranked URL for a query changes hands.

def flip_rate(weekly_df, query):
    q = weekly_df[weekly_df["query"] == query]
    winners = (
        q.sort_values("position")
         .groupby("week")
         .first()["page"]
    )
    changes = (winners != winners.shift()).sum() - 1
    weeks = max(len(winners) - 1, 1)
    return changes / weeks

A flip rate near zero means one URL owns the query and the second is harmless background noise, so you can safely ignore it. A flip rate above roughly 0.4 means Google cannot decide which page to trust, and that indecision almost always shows up as a worse average position than a single consolidated page would earn. Those are the queries to fix first.

Turning the report into action

Detection is only useful if it routes to a decision. For each confirmed query, the pipeline should recommend one of three moves. Consolidate when the two URLs serve the same intent: merge the weaker into the stronger and 301 redirect it. Differentiate when the intents genuinely differ but the pages overlap too much: rewrite one to target a distinct angle and fix the internal anchors pointing at it. Demote when one URL should never have targeted the query at all: adjust its internal links and on-page signals so it stops competing.

Feed the winner and loser URLs into your internal linking data before you act. A page that looks like a cannibalization loser is sometimes just an orphan page with almost no internal links that Google surfaced by accident, and the fix there is linking, not merging. Layering this report on top of your link graph prevents you from redirecting a page that simply needed support.

Running it on a schedule

Cannibalization is not a one-time cleanup. New posts compete with old ones the moment they publish, so this belongs in the same weekly job that watches for content decay and seasonal decline. Run the query-and-page pull, regenerate the report, diff it against last week, and alert only on newly cannibalized queries above your volume threshold. That diff is what keeps the output actionable instead of a 400-row spreadsheet nobody reads.

Store each week’s report so you can measure whether your consolidations actually worked. After a merge and redirect, the surviving URL’s average position for the query should improve and its flip rate should collapse toward zero. If it does not, the intents were more distinct than you assumed, and the correct move was differentiation all along.

Frequently asked questions

Is keyword cannibalization always bad?

No. Two URLs ranking for the same query only hurts when they trade positions and split clicks. If one URL owns the query and the other appears rarely at a much worse position, there is no meaningful competition and no action is needed.

Why use Search Console instead of a rank tracker?

A rank tracker shows the single URL ranking on the day it checked. Search Console records every URL Google surfaced for a query across the whole period, which is the only way to see the impression split and the week-to-week flips that define real cannibalization.

How much data do I need before the report is reliable?

Use at least 28 days so a single volatile week does not skew the impression shares, and ignore queries below a minimum impression floor. Low-volume queries produce unstable ratios that generate false positives.

Should I always redirect the losing page?

Only when both pages serve the same intent. If the intents differ, redirecting destroys a page that could rank on its own once you differentiate it and fix its internal links. Confirm intent before consolidating.