Build an LLM Content-Quality Scorer Aligned to Google’s Quality Rater Guidelines (Python)

Google’s Quality Rater Guidelines (QRG) do not rank a single page. But they describe the target that Google’s ranking systems are trained to approximate. Human raters read pages and assign Page Quality and Needs Met ratings, and those judgments become calibration data for the models that actually order search results. If you run a large site, you cannot hire an army of raters to audit thousands of URLs. You can, however, approximate their rubric with an LLM and score your library at scale.

This article builds a defensible content-quality scorer in Python: a rubric mapped to real QRG concepts, a structured prompt that forces evidence-bound JSON output, a calibration step against a human-labeled seed set, and a refresh queue ranked by quality gap multiplied by traffic opportunity. The goal is triage you can trust, not a magic ranking oracle.

What the Quality Rater Guidelines actually measure

The QRG evaluate two largely independent axes. Needs Met measures how well a result satisfies a specific query and its likely intent. Page Quality measures the page on its own terms: the clarity of its purpose, the effort and skill in the main content, the expertise of the creator, the reputation of the site, and how trustworthy the page is. E-E-A-T (Experience, Expertise, Authoritativeness, Trust) sits inside Page Quality, with Trust as the most important member of the family.

This distinction matters for automation. You cannot reproduce Needs Met without a query, and most pages rank for dozens of them. But Page Quality is a per-URL property you can approximate from the page alone. So the scorer we build targets Page Quality and deliberately stays silent on query-level relevance. One honest caveat up front: raters do not set rankings. They produce training signal. An LLM imitating a rater is therefore a proxy of a proxy. Treat the output as a prioritization aid, never as a verdict.

Design the rubric before you touch an LLM

The most common mistake is asking a model to “rate this page from 1 to 10.” That produces confident, uncalibrated noise. Instead, decompose Page Quality into observable sub-signals, each scored on a short scale with explicit anchors. Explicit anchors are what stop the model from drifting between runs.

A rubric that maps cleanly onto the QRG uses six dimensions, each scored 0 to 3: Purpose clarity (is the page’s goal obvious and beneficial?), Main content quality and effort (originality, depth, and skill versus filler and paraphrase), Expertise signals (does the content demonstrate first-hand experience or domain knowledge?), Reputation surface (visible author, credentials, about/contact information), Trust (accuracy, citations, current dates, no deceptive patterns), and YMYL risk (does the topic touch money, health, or safety, raising the bar?). Anchor each score. For Main content, for example: 0 = auto-generated or copied, 1 = thin rewrite of common knowledge, 2 = competent and accurate but unremarkable, 3 = original insight, data, or experience a competitor cannot easily copy.

Extract clean main content first

Feed the model the article, not the chrome. Navigation, cookie banners, and footer link farms will pollute a quality judgment badly. Extract the main content and metadata before scoring.

import trafilatura

def extract(url: str) -> dict:
    downloaded = trafilatura.fetch_url(url)
    text = trafilatura.extract(
        downloaded, include_comments=False, include_tables=True
    )
    meta = trafilatura.extract_metadata(downloaded)
    return {
        "url": url,
        "text": (text or "")[:12000],   # cap tokens; score the body, not the archive
        "author": getattr(meta, "author", None),
        "date": getattr(meta, "date", None),
        "title": getattr(meta, "title", None),
    }

The scoring prompt: force structured, evidence-bound output

Give the model the full rubric in the system message, set temperature to 0, and require JSON with a per-dimension score, a one-sentence justification quoting evidence from the page, and a confidence flag. Crucially, include an “insufficient evidence” escape so the model abstains instead of inventing an author’s credentials.

from pydantic import BaseModel, conint
import json, openai

class Dim(BaseModel):
    score: conint(ge=0, le=3)
    evidence: str          # a short quote or paraphrase from the page
    confident: bool

class PageScore(BaseModel):
    purpose: Dim
    main_content: Dim
    expertise: Dim
    reputation: Dim
    trust: Dim
    ymyl_risk: conint(ge=0, le=3)

RUBRIC = open("rubric.md").read()   # the anchors defined above

def score_page(page: dict) -> PageScore:
    msg = [
        {"role": "system", "content":
            "You are a search quality rater. Apply this rubric strictly. "
            "Quote evidence from the page. If a dimension cannot be judged "
            "from the text, set confident=false and score conservatively." + RUBRIC},
        {"role": "user", "content":
            "URL " + page["url"] + " AUTHOR " + str(page["author"]) + " " + page["text"]},
    ]
    resp = openai.chat.completions.create(
        model="gpt-4o-mini", temperature=0, messages=msg,
        response_format={"type": "json_object"},
    )
    return PageScore(**json.loads(resp.choices[0].message.content))

Requiring an evidence quote per dimension is not decoration. It forces the model to ground each score in the text and gives a human reviewer a fast way to audit disagreements later.

Calibrate against a human-labeled seed set

An uncalibrated scorer is a random number generator with good manners. Before you trust a single number, hand-label 50 to 100 URLs across the quality spectrum yourself, using the same rubric. Then run the LLM on the same set and measure agreement with quadratic weighted Cohen’s kappa, which rewards near-misses and punishes wild disagreement.

from sklearn.metrics import cohen_kappa_score

human = [ ]   # your labels, 0-3 per dimension
model = [ ]   # llm scores on the same URLs
kappa = cohen_kappa_score(human, model, weights="quadratic")
print("agreement:", round(kappa, 2))   # aim for > 0.6 before trusting at scale

If agreement is weak, the fix is almost always the rubric, not the model: sharpen the anchors, add two or three few-shot examples of borderline pages, or split an overloaded dimension. LLMs also show a well-documented leniency bias, clustering scores toward the top of a scale, so watch for a model that never awards a 0. Running each page three times and taking the median tames run-to-run variance at triple the cost.

Aggregate to a page score and a refresh queue

Collapse the dimensions into a weighted composite. For YMYL pages, weight Trust and Expertise more heavily, mirroring how the guidelines demand a higher bar there. Then, and this is the step that turns a spreadsheet into a plan, multiply the quality gap by traffic opportunity pulled from Search Console.

def composite(s: PageScore) -> float:
    w = {"purpose":1, "main_content":3, "expertise":2, "reputation":1, "trust":3}
    if s.ymyl_risk >= 2:            # raise the bar on money/health/safety topics
        w["trust"], w["expertise"] = 5, 3
    total = sum(getattr(s, k).score * v for k, v in w.items())
    return total / (3 * sum(w.values()))     # normalize 0..1

# priority = how bad it is  x  how much traffic it could earn
priority = (1 - composite(score)) * gsc_impressions[url]

A weak page nobody sees can wait. A weak page already collecting impressions is a rankings upgrade waiting to happen. This is the same logic behind a good content decay refresh queue, and the two pipelines share an output: a ranked list of URLs worth a human’s afternoon.

Pitfalls that make the score lie

Four failure modes deserve permanent suspicion. Circularity: using an LLM to grade LLM-written content invites the model to reward its own stylistic tics; keep a human in the loop on the bottom decile. Prompt sensitivity: reword the rubric and scores shift, which is exactly why calibration is not optional. Cost and drift: triage the whole library with a cheap model, escalate only borderline pages to an expensive one, and pin the model version because a silent upgrade will re-baseline your scores overnight. Automation overreach: never auto-noindex on score alone. As the argument that word count will not save you from quality penalties makes clear, quality is qualitative; the scorer’s job is to point a human at the right pages, and a broader recovery plan is where those judgments turn into ranking gains.

Frequently asked questions

Can an LLM really replace human quality raters?

No, and it should not try to. Human raters generate the ground truth that ranking models learn from. An LLM scorer is a triage tool that lets a small team focus its limited attention on the pages most likely to be dragging quality down. Its value is prioritization at scale, not final judgment.

Which model should I use for scoring?

Use a cheap, fast model to score the entire library, then escalate only the borderline or high-traffic pages to a stronger model. Pin the exact model version so a provider-side upgrade does not silently shift your baseline, and re-run your calibration set whenever you change models.

How many pages do I need to hand-label first?

Fifty to one hundred URLs spread across the quality spectrum is usually enough to compute a meaningful weighted kappa. Aim for agreement above 0.6 before trusting the scorer at scale, and re-label a fresh sample periodically to catch drift.

Is this safe to run on YMYL pages?

Score them, but weight Trust and Expertise more heavily and never act automatically on the result. Money, health, and safety topics carry the highest bar in the guidelines and the highest cost when a machine gets it wrong, so a human should review every YMYL change the scorer proposes.