Build an End-to-End Keyword Research Pipeline in Python: Seeds, Autocomplete, and People Also Ask

Most keyword research still starts inside a paid tool, where you type a seed, export a CSV, and trust a black-box “difficulty” score you cannot audit. That works until you need thousands of queries across dozens of topics, or until you want signals the tools do not expose — the exact questions real users type into Google, and the phrasing autocomplete surfaces before anyone has clicked. A pipeline you own fixes both problems. It runs on a schedule, pulls directly from the surfaces searchers see, and lets you define opportunity on your terms instead of a vendor’s.

This guide builds that pipeline in Python. It harvests seed keywords, expands them with Google Autocomplete, mines People Also Ask (PAA) questions, then deduplicates and scores everything into a prioritized worklist. Every stage is code you can read, modify, and version-control.

Why build the pipeline instead of buying another subscription

Keyword tools are convenient, but they hide their sources and normalize their numbers so every customer sees the same suggestions. Two consequences follow. First, the “opportunity” you find is the opportunity everyone with the same subscription found, which is why so many content calendars converge on identical head terms. Second, you cannot inspect why a keyword scored the way it did, so you cannot tune the logic for your niche.

Autocomplete and PAA are different. They are live reflections of demand, refreshed constantly, and they expose long-tail phrasing and question intent that aggregated databases lag on by months. Harvesting them yourself means your seed list becomes a branching tree of real queries, and your scoring reflects your own site’s authority and margins rather than a generic index. The trade-off is that you are responsible for rate limits and hygiene — covered near the end.

The architecture: four composable stages

Keep each stage independent so you can rerun or replace one without touching the others. The flow is: seeds → autocomplete expansion → PAA mining → dedupe and score. Each stage writes a plain data structure (a list of dicts) that the next stage consumes, which makes the whole thing easy to test and to cache between runs.

Stage 1 — Harvest seed keywords

Seeds are the roots of the tree. The strongest source is your own Google Search Console data, because those are terms you already have a footprint for. Pull queries where you rank on pages two or three — the ones with impressions but weak position, where a targeted page can move the needle fastest.

import pandas as pd

def seeds_from_gsc(df, min_impressions=50):
    """df has columns: query, impressions, position."""
    opportunity = df[
        (df["impressions"] >= min_impressions) &
        (df["position"].between(11, 30))
    ]
    return (
        opportunity.sort_values("impressions", ascending=False)
        ["query"].str.lower().unique().tolist()
    )

seeds = seeds_from_gsc(gsc_df)[:200]

No Search Console history for a new site? Seed from competitor headings, your product taxonomy, or a short list of hand-written topics. The pipeline does not care where seeds come from — it only cares that they are lowercase, deduplicated strings.

Stage 2 — Expand with Google Autocomplete

Google exposes autocomplete through a lightweight suggestion endpoint that returns JSON. For each seed, you can also append each letter of the alphabet to force deeper branches — this is the classic “alphabet soup” technique that multiplies one seed into dozens of concrete phrases.

import requests, time, string

def autocomplete(term, lang="en", country="us"):
    url = "https://suggestqueries.google.com/complete/search"
    params = {"client": "firefox", "q": term, "hl": lang, "gl": country}
    try:
        r = requests.get(url, params=params, timeout=10)
        r.raise_for_status()
        return r.json()[1]
    except (requests.RequestException, ValueError):
        return []

def expand_seed(seed):
    found = set(autocomplete(seed))
    for letter in string.ascii_lowercase:
        found.update(autocomplete(seed + " " + letter))
        time.sleep(0.4)  # be gentle
    return found

expanded = {}
for s in seeds:
    expanded[s] = expand_seed(s)
    time.sleep(1)

A single seed like “keyword research” now branches into “keyword research for youtube”, “keyword research free tools”, “keyword research vs competitor analysis”, and so on. You have turned 200 seeds into thousands of candidate phrases without touching a paid API.

Stage 3 — Mine People Also Ask questions

Autocomplete gives you phrases; PAA gives you the questions Google itself considers related, which map cleanly onto H2s, FAQ blocks, and featured-snippet targets. PAA lives in the SERP, so you fetch a results page and parse the question boxes. In production, route this through a SERP API or a headless browser to stay reliable — the same infrastructure decision covered in our teardown of SERP scraping approaches at scale. The parsing logic is what matters here:

from bs4 import BeautifulSoup

def parse_paa(html):
    soup = BeautifulSoup(html, "html.parser")
    questions = []
    for node in soup.select("[data-q], div[role='heading']"):
        text = node.get("data-q") or node.get_text(strip=True)
        if text and text.endswith("?") and len(text) > 12:
            questions.append(text.lower())
    return list(dict.fromkeys(questions))

def paa_for(query, fetch_serp):
    return parse_paa(fetch_serp(query))

Run PAA mining only on your highest-value expanded phrases, not every candidate — SERP fetches are the expensive part of the pipeline. A good rule is to mine PAA for the top few hundred phrases after the scoring step, then feed the returned questions back in as new seeds for a second pass. That feedback loop is where a keyword list becomes a keyword map.

Stage 4 — Deduplicate and score by opportunity

Raw expansion produces heavy overlap: “best crm software”, “the best crm software”, and “best crm software 2026” are near-duplicates. Collapse them by normalizing whitespace and stopwords, then fingerprinting. This is a lightweight cousin of the technique in our guide to detecting near-duplicate pages with MinHash, scaled down to short strings.

import re

STOP = {"the", "a", "an", "for", "to", "of", "in", "best", "vs"}

def fingerprint(kw):
    tokens = re.sub(r"[^a-z0-9 ]", "", kw.lower()).split()
    core = sorted(t for t in tokens if t not in STOP)
    return " ".join(core)

def dedupe(keywords):
    seen, out = set(), []
    for kw in keywords:
        fp = fingerprint(kw)
        if fp and fp not in seen:
            seen.add(fp)
            out.append(kw)
    return out

Now score. The point of owning the pipeline is defining opportunity yourself. A simple, transparent formula multiplies demand by how well the intent fits your site and how weak your current position is:

def opportunity_score(row):
    demand = row["impressions"] ** 0.5           # dampen huge head terms
    intent = 1.4 if row["is_question"] else 1.0  # questions map to snippets
    gap = max(0, (row["position"] - 10)) / 20    # reward weak positions
    return round(demand * intent * (1 + gap), 2)

Sort descending and you have a worklist ranked by your definition of value, not a vendor’s opaque difficulty metric. Questions flagged is_question feed directly into content briefs; the clustered themes below become pillar pages.

Putting it together and scheduling

Wire the four stages into one script, cache each stage’s output to disk so a failure in PAA mining does not cost you the autocomplete run, and schedule it weekly with cron or a workflow tool. The output — a scored CSV of deduplicated keywords and questions — becomes the input to two downstream systems: grouping related terms into topics, and turning those topics into outlines. For grouping, hand the list to a keyword clustering pipeline using embeddings and HDBSCAN; for outlines, feed the top questions into an automated brief generator. The keyword pipeline is the sourcing layer beneath both.

Rate limits, ethics, and staying unblocked

Harvesting public endpoints is not a licence to hammer them. Three habits keep the pipeline sustainable. Throttle deliberately — the time.sleep calls above are the minimum, and randomized delays are better than fixed ones. Cache aggressively, because autocomplete for a given seed rarely changes within a week, so re-fetching daily is waste that only raises your block risk. And degrade gracefully: wrap every network call so a single failure returns an empty list rather than crashing a multi-hour run. For anything at real volume, put PAA and SERP fetches behind a proper SERP API so you are not managing proxies and CAPTCHAs by hand. Treat these surfaces as the shared resource they are, and the pipeline will keep running long after a scraped-until-blocked script would have died.

Frequently asked questions

Is scraping Google Autocomplete against the rules?

The suggestion endpoint is a public, unauthenticated interface, but Google’s terms discourage automated access and can rate-limit or block abusive patterns. Keep volume modest, throttle requests, cache results, and for large-scale or commercial work use a sanctioned SERP API rather than hitting the endpoint directly.

How is this different from just using a keyword tool?

Keyword tools return an aggregated, normalized database that every subscriber sees, with a difficulty score you cannot inspect. A pipeline pulls live from autocomplete and People Also Ask, exposes phrasing and questions those databases lag on, and lets you define and audit your own opportunity scoring instead of trusting a black box.

Do I need Search Console data to start?

No. Search Console is the strongest seed source because it reflects terms you already rank for, but you can seed from competitor headings, your product taxonomy, or a hand-written topic list. The expansion, mining, and scoring stages work identically regardless of where the seeds come from.

How often should I run the pipeline?

Weekly is a sensible default for most sites. Autocomplete and PAA change gradually, so daily runs mostly waste requests and raise block risk. Schedule it with cron or a workflow tool, cache each stage, and trigger an extra run when you launch a new product area or notice a shift in your Search Console queries.

What do I do with the output?

Feed the scored list into a clustering step to group terms into topics, then turn the top questions into content briefs and outlines. The keyword pipeline is the sourcing layer; clustering and brief generation are the layers that convert raw keywords into a publishable content plan.