Track SERP Features at Scale: A Python Pipeline for Featured Snippets, PAA, and AI Overviews
Rank tracking used to be simple: pull your position for a keyword, watch it move, react. But the modern Google results page is no longer a tidy list of ten blue links. A single query might render an AI Overview that answers the question before anyone scrolls, a featured snippet lifted from one lucky page, a People Also Ask accordion, a video carousel, and only then your hard-won organic listing. If your monitoring stack still reports “position 4” and nothing else, you are flying blind to the most important shifts happening in search right now.
The problem is that SERP features are volatile and query-specific. An AI Overview that triggers on a keyword today may vanish next week, and the page Google chooses to cite can change without your ranking moving an inch. To manage this at the scale of hundreds or thousands of keywords, you need an automated pipeline that captures the full anatomy of each results page on a schedule — not a human eyeballing incognito searches. This post walks through building exactly that in Python: a SERP feature tracker that records which features appear for each keyword, who owns them, and how that ownership drifts over time.
Why position-only tracking quietly fails in 2026
Consider a keyword where you rank position 3. On paper that is excellent. But if an AI Overview occupies the top of the page and synthesizes an answer from three competitors, your real estate above the fold may have collapsed even though your rank is unchanged. Conversely, owning the featured snippet on a position-1-adjacent query can be worth more clicks than the nominal top organic slot. Position is a single scalar; the SERP is a layout. Treating one as a proxy for the other leaves money on the table.
There is also a strategic layer. The set of features Google chooses to render is a signal of search intent. A query that triggers a featured snippet and a PAA block is informational and answer-hungry; a query with shopping units and no AI Overview is transactional. Tracking which features appear across your keyword universe gives you an intent map you can use to prioritize content formats — and to spot when Google reclassifies a query, which often precedes a traffic swing.
Architecture: four stages, one schedule
The pipeline breaks into four decoupled stages so that a failure in one does not corrupt the others. First, acquisition: fetch the rendered SERP HTML for each keyword. Second, parsing: extract a structured record of which features are present and which domains own them. Third, storage: append a timestamped row per keyword-feature so you can query history. Fourth, alerting: diff today against yesterday and surface meaningful changes. Keeping these separate means you can swap the acquisition layer or re-parse archived HTML without re-querying the SERP.
Stage 1 — Acquiring the SERP reliably
Scraping Google directly from a single server IP is the fastest way to get throttled and served degraded or CAPTCHA-blocked pages, which silently poison your data. For anything beyond a handful of keywords, route requests through a residential or SERP-grade proxy layer. Tools like Bright Data‘s SERP API (or any equivalent that returns geolocated, fully rendered results) exist precisely because consistent, location-accurate SERP HTML is hard to get at scale. The key principles regardless of provider: pin a consistent location and language, request the rendered page (AI Overviews are injected client-side and will be missing from a raw fetch), and add jitter between requests.
import requests, time, random
def fetch_serp(keyword, gl="us", hl="en"):
"""Return rendered SERP HTML for a keyword via a SERP-grade endpoint."""
params = {
"q": keyword,
"gl": gl, # country
"hl": hl, # language
"num": 10,
"brd_json": 0, # request raw HTML; flip to 1 for structured JSON
}
resp = requests.get(
SERP_ENDPOINT, # your proxy/SERP provider URL
params=params,
proxies=PROXY_CONFIG,
timeout=45,
)
resp.raise_for_status()
time.sleep(random.uniform(1.5, 4.0)) # be a polite citizen
return resp.textIf your provider returns structured JSON for SERP features, use it — parsing rendered Google HTML is a maintenance treadmill because the markup churns. The HTML path below is the fallback for when you only have raw pages.
Stage 2 — Parsing features into a structured record
The goal of parsing is to turn a messy page into a flat record: for each feature type, is it present, and which domain owns it? Rather than chase brittle class names, anchor on stable structural cues and visible text labels. Below is a deliberately defensive parser that degrades gracefully when one detector breaks.
from bs4 import BeautifulSoup
from urllib.parse import urlparse
def parse_serp(html, your_domain):
soup = BeautifulSoup(html, "html.parser")
text = soup.get_text(" ", strip=True).lower()
record = {
"ai_overview": False,
"featured_snippet": False,
"featured_snippet_owner": None,
"paa_present": False,
"paa_questions": [],
"you_in_top10": False,
"your_position": None,
}
# AI Overview: injected block, usually labelled in the DOM
if "ai overview" in text or soup.select_one("[data-attrid*='Overview']"):
record["ai_overview"] = True
# Featured snippet: the answer box sits above organic results
fs = soup.select_one("div.xpdopen, div[data-attrid='wa:/description']")
if fs:
record["featured_snippet"] = True
link = fs.find("a", href=True)
if link:
record["featured_snippet_owner"] = urlparse(link["href"]).netloc
# People Also Ask
paa = soup.select("div[jsname] [role='heading']")
questions = [q.get_text(strip=True) for q in paa
if q.get_text(strip=True).endswith("?")]
if questions:
record["paa_present"] = True
record["paa_questions"] = questions[:6]
# Your organic position
organic = soup.select("div.g a[href^='http']")
for i, a in enumerate(organic, start=1):
if your_domain in urlparse(a["href"]).netloc:
record["you_in_top10"] = True
record["your_position"] = i
break
return recordTreat these selectors as a starting point, not gospel — validate them against a handful of saved SERPs for your own market before trusting the output, and log the raw HTML so you can re-parse historically when Google changes its layout.
Stage 3 — Storing history so you can ask “what changed?”
The whole point is longitudinal: a single snapshot is trivia, a time series is intelligence. Append one row per keyword per run to a columnar store. A flat partitioned Parquet dataset or a small DuckDB table is plenty for tens of thousands of rows and queries instantly.
import duckdb, datetime, json
def store(keyword, record, db="serp_history.duckdb"):
con = duckdb.connect(db)
con.execute("""
CREATE TABLE IF NOT EXISTS serp_features (
captured_at TIMESTAMP, keyword VARCHAR,
ai_overview BOOLEAN, featured_snippet BOOLEAN,
featured_snippet_owner VARCHAR, paa_present BOOLEAN,
your_position INTEGER, paa_questions VARCHAR
)""")
con.execute("INSERT INTO serp_features VALUES (?,?,?,?,?,?,?,?)", [
datetime.datetime.utcnow(), keyword,
record["ai_overview"], record["featured_snippet"],
record["featured_snippet_owner"], record["paa_present"],
record["your_position"], json.dumps(record["paa_questions"]),
])
con.close()Stage 4 — Diffing and alerting on the changes that matter
Not every change deserves a notification. The signals worth waking up for are: an AI Overview newly appearing (or disappearing) on a money keyword, a featured snippet changing hands, and your page dropping out of the top ten while a feature absorbs the click. A simple self-join in SQL surfaces these by comparing each keyword’s two most recent captures.
def detect_changes(db="serp_history.duckdb"):
con = duckdb.connect(db)
return con.execute("""
WITH ranked AS (
SELECT *, ROW_NUMBER() OVER
(PARTITION BY keyword ORDER BY captured_at DESC) rn
FROM serp_features
)
SELECT a.keyword,
b.ai_overview AS was_aio, a.ai_overview AS now_aio,
b.featured_snippet_owner AS was_fs, a.featured_snippet_owner AS now_fs
FROM ranked a JOIN ranked b
ON a.keyword=b.keyword AND a.rn=1 AND b.rn=2
WHERE a.ai_overview <> b.ai_overview
OR COALESCE(a.featured_snippet_owner,'') <> COALESCE(b.featured_snippet_owner,'')
""").fetchall()Pipe the result into Slack, email, or an n8n webhook so the alert lands where your team already works. If you already run scheduled SEO jobs, this slots in next to them as one more cron entry.
What the data tells you after a few weeks
Once the history accumulates, three patterns reliably emerge. First, you will find keywords where you rank well but an AI Overview is quietly siphoning the answer — these are your highest-priority content-format rewrites, because the fix is structuring your page to be the cited source, not climbing one more rank. Second, you will catch featured snippets changing hands within a day, letting you react to a competitor’s snippet grab before a monthly audit would ever notice. Third, the presence of features becomes a leading indicator: when Google adds an AI Overview to a cluster of related queries, it is often reclassifying intent, and traffic shifts tend to follow. Teams that watch feature changes catch these inflections weeks earlier than teams watching position alone.
This is the same philosophy behind our GEO citation tracker, which monitors whether answer engines cite your site — pair the two and you cover both the classic SERP and the AI answer surfaces. To turn the opportunities this surfaces into a prioritized worklist, feed the keywords into the scoring logic from our striking-distance keyword finder. And if you would rather consume rankings from an API than scrape SERPs yourself, our breakdown of GSC vs SEMrush vs Ahrefs APIs covers the trade-offs.
Takeaways
Position is a lossy summary of a layout that grows more complex every quarter. By capturing the full feature anatomy of each SERP on a schedule, storing it as a time series, and alerting only on consequential changes, you convert a noisy, volatile results page into a structured signal you can act on. The pipeline is modest — a fetcher, a defensive parser, a columnar table, and a diff query — but it puts you ahead of the swings that position-only tracking will never show you. Start with your fifty most valuable keywords, prove the alerts are useful, and scale the keyword list from there.
Found this useful? Bookmark SEOAutomationClub and check back for weekly automation playbooks — we publish working code and real workflows, not generic listicles.
Frequently Asked Questions
Can I track AI Overviews without scraping Google myself?
Yes. Several SERP API providers return AI Overview presence and content as structured JSON, which spares you the brittle HTML parsing. Scraping yourself gives you full control and lower per-query cost at scale, but you take on proxy management and parser maintenance. For most teams under a few thousand keywords, a SERP API is the pragmatic choice; beyond that, a self-hosted scraper through a residential proxy pool becomes cost-effective.
How often should the pipeline run?
Daily is the sweet spot for most keyword sets — frequent enough to catch featured-snippet swaps and AI Overview toggles, infrequent enough to keep proxy costs and request volume reasonable. Reserve hourly runs for a small watchlist of high-value or volatile terms, and weekly for long-tail keywords where features rarely change.
Why does the parser break when I run it on raw fetched HTML?
AI Overviews and several other features are injected client-side via JavaScript and are absent from a plain HTTP response. You need a rendering layer — either a SERP provider that returns the fully rendered page, or a headless browser — for those detectors to fire. If your AI Overview detector always returns False, an un-rendered fetch is almost always the cause.
Is scraping Google SERPs allowed?
Automated querying of Google’s results sits in a gray area of its terms of service, which is why most teams use commercial SERP APIs that handle compliance and provide consistent, geolocated results. Review your provider’s terms and your own legal posture before scaling, and always rate-limit responsibly.
