Automate Title Tag & Meta Description Rewrites at Scale: A Python + LLM Pipeline for Low-CTR Pages

Most SEO teams treat title tags and meta descriptions as a set-and-forget task: write them once at publish time, then never touch them again. That’s a mistake. Your rankings move, the SERP layout changes, competitors rewrite their snippets, and the query mix that actually lands on a page drifts over months. The result is a long tail of pages that rank on page one but bleed clicks because the snippet no longer earns the click.

The fix is not a one-off audit. It’s a repeatable pipeline that finds the pages losing clicks, rewrites their titles and descriptions with an LLM under strict constraints, ships the changes, and then measures whether CTR actually improved. This post walks through that pipeline end to end, with the guardrails that keep the LLM from producing garbage and the measurement loop that separates real lift from seasonal noise.

Why CTR optimization beats chasing new rankings

Moving a page from position 8 to position 5 is expensive and slow. Improving the click-through rate of a page that already sits at position 4 is cheap, fast, and fully under your control. If a page gets 40,000 impressions a month at a 3.1% CTR, and the position-4 curve suggests it should be closer to 6.5%, you’re leaving roughly 1,300 clicks a month on the table from a single URL. Multiply that across a few hundred qualifying pages and title/description rewrites become the highest-ROI automation you can run.

The trick is targeting. You don’t rewrite everything. You rewrite pages where the expected CTR for their position materially exceeds their actual CTR. That’s the same opportunity-scoring logic behind our striking-distance keyword finder, applied to snippets instead of rankings.

Step 1 — Pull the data and score the CTR gap

Everything starts with Google Search Console. Query the Search Analytics API for the last 28 days, grouped by page, and keep clicks, impressions, CTR, and average position. Then compare each page’s actual CTR against a positional benchmark curve.

import pandas as pd
from googleapiclient.discovery import build
from google.oauth2 import service_account

SITE = "https://example.com/"
creds = service_account.Credentials.from_service_account_file(
    "gsc-service-account.json",
    scopes=["https://www.googleapis.com/auth/webmasters.readonly"],
)
gsc = build("searchconsole", "v1", credentials=creds)

resp = gsc.searchanalytics().query(siteUrl=SITE, body={
    "startDate": "2026-06-01",
    "endDate": "2026-06-28",
    "dimensions": ["page"],
    "rowLimit": 25000,
}).execute()

df = pd.DataFrame([{
    "page": r["keys"][0],
    "clicks": r["clicks"],
    "impressions": r["impressions"],
    "ctr": r["ctr"],
    "position": r["position"],
} for r in resp.get("rows", [])])

# A positional CTR benchmark curve (calibrate to YOUR site, not industry averages).
BENCHMARK = {1:0.28, 2:0.15, 3:0.10, 4:0.072, 5:0.055,
             6:0.043, 7:0.034, 8:0.028, 9:0.024, 10:0.021}

def expected_ctr(pos):
    p = min(10, max(1, round(pos)))
    return BENCHMARK[p]

df["expected_ctr"] = df["position"].apply(expected_ctr)
df["ctr_gap"] = df["expected_ctr"] - df["ctr"]
# Opportunity = how many clicks the gap is worth at current impressions.
df["opportunity"] = (df["ctr_gap"] * df["impressions"]).round(0)

candidates = df[
    (df["position"].between(2, 12)) &   # already visible, room to move
    (df["impressions"] >= 500) &         # enough volume to measure
    (df["ctr_gap"] > 0.01)               # meaningfully underperforming
].sort_values("opportunity", ascending=False).head(150)

One caveat that matters: do not use industry-average CTR curves. Brand queries, SERP features, and your niche all distort the curve. Build your own benchmark by bucketing your GSC rows by rounded position and taking the median CTR per bucket. A borrowed curve will send you rewriting pages that are actually fine.

Step 2 — Give the LLM the context it needs (not just the old title)

The quality of a rewrite depends entirely on the context you feed the model. Handing it “rewrite this title” produces bland, keyword-stuffed output. Instead, assemble a per-page brief: the current title and description, the top 5 queries the page ranks for (pulled from GSC grouped by query and page), the page’s H1, and a short extract of the intro paragraph.

def build_brief(page_url, queries, current_title, current_desc, h1, intro):
    top = ", ".join(queries[:5])
    return f"""Page: {page_url}
Current title: {current_title}
Current meta description: {current_desc}
On-page H1: {h1}
Intro excerpt: {intro[:300]}
Top queries driving impressions: {top}"""

Pulling the current on-page tags means a quick fetch-and-parse pass with requests + selectolax (or reuse your crawler’s output). The top-queries dimension is the single most valuable input — it tells the model what searchers actually typed, which is frequently different from what the page was originally optimized for.

Step 3 — Constrain the model with a strict prompt and a schema

LLMs love to exceed length limits and invent claims. Both are fatal for meta tags: Google truncates long titles, and a description that overpromises tanks trust. Force structured output and validate it in code — never trust the raw generation.

import anthropic, json

client = anthropic.Anthropic()

SYSTEM = """You are an SEO copywriter. Rewrite title tags and meta descriptions.
Rules:
- Title: 50-60 characters. Lead with the primary query intent. No clickbait, no fabricated numbers.
- Description: 140-155 characters. One concrete benefit + one reason to click. Active voice.
- Use ONLY facts present in the brief. Never invent statistics, dates, or claims.
- Include the primary query naturally; do not keyword-stuff.
Return JSON: {"title": "...", "description": "...", "rationale": "..."}"""

def rewrite(brief):
    msg = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=400,
        system=SYSTEM,
        messages=[{"role": "user", "content": brief}],
    )
    return json.loads(msg.content[0].text)

Because you asked for JSON, you can validate deterministically. This validation layer is what makes the pipeline safe to run at scale without a human eyeballing every line.

def validate(rw, primary_query):
    t, d = rw["title"], rw["description"]
    checks = {
        "title_len": 40 <= len(t) <= 62,
        "desc_len": 120 <= len(d) <= 158,
        "has_query": primary_query.lower().split()[0] in t.lower(),
        "no_pipe_spam": t.count("|") <= 1,
        "no_allcaps": not any(w.isupper() and len(w) > 3 for w in t.split()),
    }
    return all(checks.values()), checks

Anything that fails validation gets routed to a manual review queue instead of shipping. In practice, 85–90% of generations pass on the first try once the prompt is tuned; the rest are usually edge cases (branded pages, legal content) you want a human to see anyway.

Step 4 — Ship the changes and version them

How you deploy depends on your stack. On WordPress, patch the Yoast or RankMath meta fields via the REST API. On a headless or custom CMS, write to your database and trigger a rebuild. On a static site, edit front-matter and open a pull request. Whatever the target, do two things without exception:

First, store the old value before overwriting it. You need the original to measure lift and to roll back. Second, record the change date per URL — this timestamp is the anchor for your before/after analysis. A simple changes.csv with URL, old title, new title, old description, new description, and change date is enough to run the whole measurement loop. If you’re already running our content refresh pipeline, drop these rewrites into the same changelog so all your on-page edits live in one auditable place.

Step 5 — Measure the lift (this is the part everyone skips)

A rewrite that “feels better” is worthless. You need to know whether CTR actually moved, and you need to control for the fact that position and seasonality also move CTR. The clean approach is a pre/post comparison that holds position roughly constant: compare the 28 days before the change to the 28 days after, but only accept the result if average position stayed within ±0.5 across both windows. If the page also jumped from position 6 to 3, you can’t attribute the CTR gain to the snippet.

def measure_lift(page, change_date, gsc, site):
    def window_ctr(start, end):
        r = gsc.searchanalytics().query(siteUrl=site, body={
            "startDate": start, "endDate": end,
            "dimensions": ["page"],
            "dimensionFilterGroups": [{"filters": [
                {"dimension": "page", "operator": "equals", "expression": page}]}],
        }).execute().get("rows", [])
        return (r[0]["ctr"], r[0]["position"]) if r else (None, None)

    pre_ctr, pre_pos = window_ctr("2026-05-04", "2026-05-31")
    post_ctr, post_pos = window_ctr("2026-06-01", "2026-06-28")
    if None in (pre_ctr, post_ctr):
        return None
    return {
        "page": page,
        "ctr_delta": round((post_ctr - pre_ctr) * 100, 2),   # percentage points
        "pos_shift": round(post_pos - pre_pos, 2),
        "valid": abs(post_pos - pre_pos) <= 0.5,             # position held?
    }

Aggregate the valid rows and report the median CTR delta, not the mean — a couple of viral outliers will otherwise flatter your numbers. For extra rigor, hold out a control group of qualifying-but-untouched pages and compare their drift over the same window; that’s a lightweight version of the causal approach we cover in SEO split testing with Python. If your treated pages beat the control group’s drift, you have real, defensible lift.

Results you can realistically expect

On a mid-size content site, targeting the top 150 CTR-gap pages, a well-tuned pipeline typically lands a median CTR improvement of 0.4–0.9 percentage points on pages where position held steady, with roughly 15–20% of rewrites showing no meaningful change and a small fraction regressing (which is exactly why you measure and keep the rollback data). The compounding win is operational: once the pipeline exists, re-running it monthly costs minutes, and every new batch of decaying snippets gets caught automatically.

Two extensions worth building next: pipe your validated rewrites into structured data so titles and descriptions stay consistent with your automated schema markup pipeline, and add an alerting hook that flags any page whose CTR drops after a rewrite so you can roll it back within a week rather than a quarter.

Key takeaways

Title and meta description optimization is the rare SEO lever that’s high-impact, low-cost, and fully within your control. Target pages by CTR gap using your own positional benchmark, feed the LLM real query context instead of just the old tag, constrain and validate every generation in code, version your changes, and — above all — measure lift while holding position constant. Skip that last step and you’re just rewriting tags on vibes.

Found this useful? Bookmark SEO Automation Club and check back weekly — we publish a new working automation playbook every morning, always with runnable code and real workflows rather than generic listicles.

Frequently asked questions

How often should I re-run the title and meta description pipeline?

Monthly is a good cadence for most sites. CTR gaps accumulate as rankings and SERP layouts drift, so a monthly pass catches decaying snippets without over-editing. High-velocity news or e-commerce sites may benefit from a biweekly run on their highest-traffic sections.

Will Google penalize AI-generated meta descriptions?

No. Google’s guidance targets low-value, unhelpful content at scale, not the tool used to write a snippet. As long as the descriptions are accurate, non-deceptive, and genuinely reflect the page, LLM-assisted rewrites are fine. The validation layer that blocks fabricated claims is what keeps you on the right side of that line.

Does Google always use the title and description I set?

Not always — Google rewrites titles a meaningful share of the time and frequently generates its own description from page content to match the query. That’s actually an argument for measurement: track which of your rewrites Google keeps, and focus future effort on the pages and query types where your tags stick.

Can I run this without the Anthropic API?

Yes. The pipeline is model-agnostic — swap the rewrite function for any capable LLM (OpenAI, a self-hosted Llama, or Gemini). The structured-output-plus-validation pattern is what matters, not the specific provider. Keep the JSON schema and the deterministic checks identical so results stay comparable across models.