Automate SEO Content Briefs at Scale: A Python Pipeline That Reads the SERP

Every content team burns hours on the same ritual: open the target query, squint at the top ten results, tab through each one, jot down the headings, guess at the entities Google seems to reward, and hand the writer a Google Doc that’s already stale by the time it’s read. It’s the least creative part of content production and the easiest to get wrong. It’s also, conveniently, the part a machine does better than a human.

This post walks through a Python pipeline that turns a raw SERP into a structured editorial brief — heading skeleton, entity checklist, question targets, and word-count band — in the time it takes to make coffee. It’s not a “paste your keyword into an AI tool” trick. It’s a repeatable system you own, running on the same SERP data Google is actually ranking, that you can point at 5 queries or 500. I’ll show the architecture, the code for each stage, and the guardrails that keep it from producing the thin, templated garbage Google’s scaled-content policies exist to punish.

Why a brief pipeline beats a brief prompt

The tempting shortcut is to ask an LLM “write me a content brief for keyword.” The output looks plausible and is quietly useless: the model is inventing what it thinks ranks, not observing what does. A brief is only as good as its grounding, and the ground truth is the live SERP for your target market, on your target device, today.

A pipeline fixes three failure modes at once. It grounds every recommendation in the actual top-ranking pages rather than the model’s memory. It’s deterministic where it should be — heading extraction and entity frequency are counting problems, not creative ones — and creative only where judgment adds value. And it scales linearly: the marginal cost of the 200th brief is an API call, not an afternoon. The design principle is simple: scrape facts, compute structure, let the LLM synthesize only the last mile.

The architecture in four stages

The pipeline is a straight line with four stages, each writing to a JSON artifact the next stage reads. Keeping the stages decoupled means you can swap the SERP source, re-run entity extraction with a better model, or regenerate the outline without re-scraping — and you can inspect exactly where a bad brief went wrong.

Stage one fetches the SERP for the query. Stage two scrapes the top-ranking URLs and pulls their heading trees and body text. Stage three computes the shared structure — heading patterns, entity frequencies, and the questions competitors answer. Stage four hands that structured evidence to an LLM to assemble the brief. Nothing in stages one through three requires a model; they’re plain data engineering, which is exactly why they’re reliable.

Stage 1 — Pull the live SERP

You need the ranking URLs for your exact query, locale, and device. Scraping Google directly is brittle and against the grain; use a SERP API instead. Bright Data’s SERP API, the free tier of a rank-data provider, or your own rank-tracking stack all work — the only contract this stage needs to honor is “return an ordered list of result URLs.”

import requests

def fetch_serp(query, gl="us", hl="en", num=10):
    """Return ordered organic result URLs for a query."""
    resp = requests.get(
        "https://api.brightdata.com/serp/req",
        params={"q": query, "gl": gl, "hl": hl, "num": num},
        headers={"Authorization": f"Bearer {SERP_TOKEN}"},
        timeout=30,
    )
    resp.raise_for_status()
    data = resp.json()
    return [r["link"] for r in data["organic"] if "link" in r][:num]

Keep the locale explicit. A brief built on gl=us results is the wrong brief for a UK page, and the difference in the top ten is often larger than teams expect.

Stage 2 — Scrape headings and body from each result

For each URL, fetch the HTML and extract the heading tree plus the visible body text. The heading tree is the structural signal; the body text feeds entity extraction in the next stage. Skip anything that isn’t a genuine content page — PDFs, video results, and forum threads distort the structure you’re trying to model.

from bs4 import BeautifulSoup

def scrape_page(url):
    html = requests.get(url, timeout=30,
                        headers={"User-Agent": "Mozilla/5.0"}).text
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "nav", "footer"]):
        tag.decompose()
    headings = [
        {"level": int(h.name[1]), "text": h.get_text(strip=True)}
        for h in soup.find_all(["h1", "h2", "h3"])
        if h.get_text(strip=True)
    ]
    body = " ".join(p.get_text(" ", strip=True)
                    for p in soup.find_all("p"))
    return {"url": url, "headings": headings, "body": body}

Wrap this in a try/except and a short concurrency pool. Two or three of your ten targets will time out, block the request, or hide their content behind JavaScript; the pipeline should log and skip them, not crash. A brief built on seven clean competitors is fine.

Stage 3 — Compute the shared structure

This is the stage that separates a real brief from a hallucinated one, and it involves no AI at all — just counting. Three signals come out of it.

Heading consensus. Normalize every H2/H3 (lowercase, strip stopwords) and cluster near-duplicates so “how much does it cost” and “pricing” collapse into one theme. Any theme that appears on a majority of pages is a section your brief must require. This is the same clustering logic behind a keyword clustering pipeline, applied to headings instead of queries.

from collections import Counter

def heading_consensus(pages, min_share=0.5):
    n = len(pages)
    themes = Counter()
    for page in pages:
        seen = {normalize(h["text"]) for h in page["headings"]
                if h["level"] in (2, 3)}
        themes.update(seen)
    return [theme for theme, c in themes.items()
            if c / n >= min_share]

Entity coverage. Run the pooled body text through a named-entity model (spaCy’s en_core_web_trf is plenty) and rank entities by how many separate competitors mention them. An entity that shows up on eight of ten pages is table stakes; one that shows up on two is an opportunity or a distraction. Ranking by document frequency, not raw count, keeps one verbose competitor from skewing the list.

Question targets. Harvest the interrogative headings and any “People Also Ask” questions from the SERP payload. These become your FAQ block — and, downstream, your FAQ schema, which pairs naturally with a schema markup pipeline so the questions you answer are also the questions you mark up.

Stage 4 — Assemble the brief with the LLM

Only now does a model enter, and its job is deliberately narrow: take the computed structure and write it up as a coherent outline with guidance. You are not asking it to decide what ranks — stage three already did, from evidence. You’re asking it to sequence the required sections logically, phrase the H2s in natural language, and note the angle each section should take.

brief_prompt = f"""
You are drafting an SEO content brief. Use ONLY the evidence below.
Do not invent sections that aren't supported by the data.

Target query: {query}
Required themes (on majority of ranking pages): {consensus}
Entities to cover (ranked by competitor coverage): {entities[:25]}
Questions to answer: {questions}
Median competitor word count: {median_wordcount}

Produce: a title suggestion, an H2/H3 outline where every required
theme maps to a heading, a one-line intent note per section, the
entity checklist, and a target word-count band (median +/- 15%).
"""

Feeding the median competitor word count matters. It anchors the writer to the length the query actually rewards instead of an arbitrary “aim for 2,000 words” rule. If the top ten average 900 words, a 2,500-word brief is a signal you’ve misread intent.

Guardrails against scaled-content abuse

Here is where teams get themselves in trouble. The moment you can generate 500 briefs, someone will want to auto-generate 500 articles from them and publish. Google’s March 2024 scaled content abuse policy exists precisely to demote that behavior, and it doesn’t care whether a human or a model pressed the button — it cares whether the output is unhelpful, templated, and made primarily for rankings.

Three rules keep the pipeline on the right side of the line. First, the brief is an input to a human writer, not a draft. The system organizes evidence; a person supplies the first-hand experience, opinion, and original data that a scraped SERP by definition cannot contain. Second, reject briefs where the entity and question sets are near-identical across a keyword cluster — that’s the pipeline telling you those queries are one page, not ten, and you should consolidate rather than spin up thin variations. Third, cap velocity to what your team can genuinely enrich. A brief you can’t staff with real expertise is a liability, not an asset.

The pipeline’s value is compression of grunt work, not replacement of judgment. It hands the writer a map. The writer still has to walk the territory — and it’s the walking, the first-hand detail, that Google’s helpful-content systems reward and that a competitor with the same SERP data can’t copy.

Results and what to measure

In practice, teams running a pipeline like this report brief-production time dropping from roughly 45–60 minutes of manual SERP analysis to a two-to-three-minute pipeline run plus a ten-minute human review. The more interesting effect is consistency: every brief now covers the majority-consensus themes, so the “we forgot the pricing section and lost the featured snippet” class of mistake disappears.

Track three things. Coverage — the share of consensus themes your published piece actually includes, which should approach 100%. Time-to-brief, to prove the efficiency case. And, downstream, the ranking delta for pages built from pipeline briefs versus manual ones, ideally isolated the way you’d run any opportunity-scored content experiment. If the pipeline isn’t moving positions after a fair window, the problem is upstream — intent mismatch or thin enrichment — and the artifacts from each stage tell you exactly where.

Where to take it next

The natural extension is to close the loop with your own performance data: after a page ranks, feed its Search Console performance back into a refresh brief so the same architecture that scoped the page also maintains it. Beyond that, you can layer SERP-feature detection into stage one (does this query trigger an AI Overview? a video pack?) so the brief accounts for what winning the page actually requires, not just the blue links.

If you build automation systems like this, bookmark the club — we publish a working SEO automation playbook every week, code included, no fluff.

Frequently asked questions

Do I need a paid SERP API to run this?

No. Any source that returns ordered ranking URLs works — a free-tier rank data provider, an existing rank tracker, or a self-hosted SERP scraper. The pipeline treats stage one as a black box that returns a list of URLs, so you can start free and swap in a paid API only when you need scale or reliability.

Won’t this produce duplicate, templated content that Google penalizes?

Only if you misuse it. The pipeline generates briefs, not articles. Google’s scaled content abuse policy targets mass-produced unhelpful pages; a brief that a human writer enriches with first-hand experience and original insight is the opposite of that. The guardrails in this post — human enrichment, cluster consolidation, and velocity caps — exist specifically to keep the output helpful.

How is this different from asking ChatGPT to write a brief?

Grounding. A raw LLM prompt produces what the model imagines ranks, from stale training data. This pipeline computes the brief from the pages that actually rank right now — real heading consensus, real entity frequencies, real competitor word counts — and uses the model only to write up that evidence, not to invent it.

How many competitor pages should I analyze per brief?

The top ten organic results is the standard, but expect to lose two or three to timeouts, paywalls, or JavaScript-rendered content. A brief built on the seven or eight you can cleanly scrape is reliable. Consensus thresholds (majority of pages) self-correct for a smaller sample, so you don’t need all ten to trust the structure.