Automate XML Sitemap Auditing: A Python Pipeline to Catch the Non-Indexable URLs Wasting Your Crawl Budget

Your XML sitemap is the one file where you get to tell Google, explicitly, “these are the pages I want you to crawl and index.” So it is strange how many sitemaps quietly become a liability instead of an asset. The typical WordPress or headless sitemap is auto-generated, never audited, and slowly fills up with URLs that redirect, return 404, carry a noindex, or canonicalize somewhere else. Every one of those is a mixed signal you are sending to a crawler that has a finite budget for your site.

Google has said for years that it treats lastmod and sitemap inclusion as hints, not commands. That is true, but it undersells the damage a dirty sitemap does. When a meaningful fraction of your submitted URLs are non-indexable, you are training Googlebot to distrust the file. The URLs you actually care about get lumped in with the noise. This post walks through a Python pipeline that audits a sitemap the way a skeptical crawler would, and produces a triage list you can act on the same afternoon.

What a healthy sitemap actually contains

Here is the rule I hold sitemaps to, and it is stricter than most CMS defaults: a URL belongs in your sitemap if and only if it returns 200, is self-canonical, is not blocked by noindex or robots.txt, and is a page you would be happy to see rank. That is the entire test. Every URL that fails it should be removed, not because Google will penalize you, but because each bad entry dilutes the signal and burns crawl requests on pages that will never earn a spot in the index.

The failure modes cluster into a handful of categories, and it is worth naming them because your audit will bucket URLs into exactly these:

• Non-200 responses. Redirected URLs (301/302) and dead URLs (404/410) that the CMS forgot to drop when a slug changed. Redirect chains are especially common after a migration.
• Canonicalized elsewhere. The URL returns 200 but its rel=canonical points to a different URL. You are asking Google to crawl a page that itself says “index that other one instead.”
• Noindex present. A meta robots noindex or X-Robots-Tag: noindex header on a URL you are simultaneously submitting for indexing. This is the most self-contradictory signal of all.
• Robots-blocked. The URL is disallowed in robots.txt, so Googlebot cannot even fetch it to see the sitemap made a promise it cannot keep.
• Structural bloat. Tag archives, paginated series, filtered/faceted parameter URLs, and attachment pages that the sitemap plugin swept in by default.

In audits I have run, the second and third categories are the sneakiest, because the URLs return 200 and look fine to a casual check. You only catch them by fetching the page and reading the canonical and robots signals, which is exactly what the pipeline below does.

The pipeline in four stages

The design is deliberately simple: expand the sitemap into a flat URL list, fetch each URL, extract the indexability signals, and classify. No database and no crawl of the whole site required for the first pass — the sitemap is your URL list.

Stage 1: Expand the sitemap (including sitemap indexes)

Real sites rarely have one sitemap. They have a sitemap index that points to a dozen child sitemaps. Your parser has to recurse. It also has to respect the two hard limits Google enforces: 50,000 URLs and 50 MB uncompressed per file. If a single sitemap exceeds either, that is itself a finding.

import gzip, requests
from urllib.parse import urljoin
from lxml import etree

HEADERS = {"User-Agent": "SitemapAudit/1.0 (+https://example.com)"}
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def fetch(url):
    r = requests.get(url, headers=HEADERS, timeout=30)
    r.raise_for_status()
    body = r.content
    if url.endswith(".gz"):
        body = gzip.decompress(body)
    return body

def expand(url, seen=None):
    """Yield (loc, lastmod) tuples from a sitemap or sitemap index, recursively."""
    seen = seen or set()
    if url in seen:
        return
    seen.add(url)
    root = etree.fromstring(fetch(url))
    tag = etree.QName(root).localname
    if tag == "sitemapindex":
        for sm in root.findall("sm:sitemap", NS):
            child = sm.findtext("sm:loc", namespaces=NS)
            yield from expand(child, seen)
    else:  # urlset
        for u in root.findall("sm:url", NS):
            loc = u.findtext("sm:loc", namespaces=NS)
            lastmod = u.findtext("sm:lastmod", namespaces=NS)
            yield loc, lastmod

This is also where you learn things the CMS never told you. Print len(list(expand(root_sitemap))) and compare it to the number of published posts and pages you think you have. A store with 900 products and a sitemap of 14,000 URLs has a faceted-navigation problem hiding in plain sight — the same crawl-budget drain you would otherwise only spot in server log file analysis.

Stage 2: Fetch each URL and read the real signals

Indexability lives in three places, and a thorough audit checks all three because they can disagree: the HTTP status and headers, the X-Robots-Tag header, and the <meta name="robots"> tag in the HTML. Do not follow redirects automatically — you want to record that the URL redirected, and to where.

from lxml import html as lxml_html

def inspect(url):
    r = requests.get(url, headers=HEADERS, timeout=30, allow_redirects=False)
    signals = {
        "url": url,
        "status": r.status_code,
        "final_url": r.headers.get("Location") if r.is_redirect else url,
        "x_robots": r.headers.get("X-Robots-Tag", "").lower(),
        "canonical": None,
        "meta_robots": "",
    }
    if r.status_code == 200 and "text/html" in r.headers.get("Content-Type", ""):
        doc = lxml_html.fromstring(r.content)
        can = doc.xpath("//link[@rel='canonical']/@href")
        signals["canonical"] = urljoin(url, can[0]) if can else None
        mr = doc.xpath("//meta[@name='robots']/@content")
        signals["meta_robots"] = (mr[0] if mr else "").lower()
    return signals

One caveat worth stating plainly: this reads the raw HTML response, not the rendered DOM. If your canonical or robots tag is injected by JavaScript, requests will not see it, and you will get false positives. That is a real gap on client-rendered sites, and the fix is to swap the fetch for a headless browser — the same rendering-parity concern I covered in the Playwright rendering-parity pipeline. For server-rendered WordPress, the raw fetch is accurate.

Stage 3: Classify each URL against the rule

Now apply the one-line test from earlier and assign every URL a verdict. The order of the checks matters: a URL can fail several tests at once, and you want to report the most actionable reason first.

def classify(s):
    if s["status"] in (301, 302, 303, 307, 308):
        return "REDIRECT", f"-> {s['final_url']}"
    if s["status"] >= 400:
        return "ERROR", f"status {s['status']}"
    if "noindex" in s["x_robots"] or "noindex" in s["meta_robots"]:
        return "NOINDEX", "noindex directive present"
    if s["canonical"] and s["canonical"].rstrip("/") != s["url"].rstrip("/"):
        return "NON_CANONICAL", f"canonical -> {s['canonical']}"
    return "OK", "indexable and self-canonical"

Stage 4: Report the triage list

The output you want is not a wall of green “OK” rows. It is a short list of the URLs that violate the rule, grouped by category, sorted so the cheapest wins are on top. A concurrent fetch keeps even a 10,000-URL sitemap under a few minutes.

from concurrent.futures import ThreadPoolExecutor
from collections import Counter, defaultdict

urls = [loc for loc, _ in expand(ROOT_SITEMAP)]
with ThreadPoolExecutor(max_workers=10) as pool:
    results = list(pool.map(inspect, urls))

buckets = defaultdict(list)
for s in results:
    verdict, detail = classify(s)
    buckets[verdict].append((s["url"], detail))

summary = Counter({k: len(v) for k, v in buckets.items()})
print("Sitemap health:", dict(summary))
for verdict in ("ERROR", "REDIRECT", "NOINDEX", "NON_CANONICAL"):
    for url, detail in buckets.get(verdict, [])[:50]:
        print(f"[{verdict}] {url}  {detail}")

Run this on a schedule — weekly is plenty for most sites — and diff the summary counts against the last run. A sudden jump in NOINDEX or REDIRECT usually means a plugin update or a migration changed URL behavior without anyone touching the sitemap. That early-warning property is the whole point, and it pairs naturally with a per-URL index coverage watchdog built on the GSC URL Inspection API.

The reverse problem: pages missing from the sitemap

Everything above finds bad URLs that are in the sitemap. The mirror-image problem is good URLs that are not. A page that is indexable and internally linked but absent from the sitemap loses a direct discovery signal. To catch these you need a second source of truth — a full crawl or the set of indexable URLs from GSC — and a set difference against the sitemap membership. That is close cousin to orphan page detection, and in practice I run both from the same crawl output: one set operation finds URLs crawled-but-not-in-sitemap, the other finds URLs in-sitemap-but-not-linked.

What to do with the findings

Resist the urge to hand-edit the sitemap. The fix almost always belongs upstream. If tag archives are bloating the file, tell the SEO plugin to exclude tag and attachment URLs. If redirected URLs keep reappearing, the CMS is regenerating the sitemap from a stale query — fix the query, not the file. The sitemap should be a derived artifact of your true set of canonical, indexable pages, and your job as an automator is to keep that derivation honest. When the pipeline reports zero URLs in the ERROR, REDIRECT, NOINDEX, and NON_CANONICAL buckets, you have a sitemap that says exactly one thing to Google, clearly.

Frequently asked questions

Does having non-indexable URLs in my sitemap cause a Google penalty?

No. There is no direct penalty. The harm is indirect: a sitemap full of redirects, errors, and noindex URLs teaches Google to trust the file less, and it wastes crawl requests on pages that cannot be indexed. On large sites where crawl budget matters, that waste is measurable. On small sites the main cost is that Search Console’s “Discovered — currently not indexed” and coverage reports get noisier and harder to read.

How often should I run a sitemap audit?

Weekly is enough for most sites, run as a scheduled job that diffs the category counts against the previous run. Run it immediately after any migration, bulk URL change, theme or SEO-plugin update, or platform move, since those are the events most likely to silently break the mapping between your sitemap and your live URLs.

Will this pipeline work if my canonical tags are added by JavaScript?

Not reliably. The example uses a plain HTTP fetch, which reads the raw HTML and cannot see tags injected after render. On client-rendered sites you will get false positives for missing or wrong canonicals. Replace the fetch step with a headless browser like Playwright so you audit the rendered DOM instead of the initial response.

Should redirected URLs stay in the sitemap until Google processes the redirect?

No. Once a URL 301-redirects, its replacement should be in the sitemap and the old URL removed. Keeping the old URL asks Google to crawl a hop it does not need. Submit the destination URL directly so the crawler reaches the live page in one request.

What is the difference between this and Search Console’s sitemap report?

Search Console tells you how many submitted URLs it discovered and broadly how many are indexed, but it does not itemize why specific URLs are non-indexable or list the exact canonical and robots conflicts per URL. This pipeline gives you that per-URL detail on demand, on your schedule, and lets you diff results over time to catch regressions the day they appear rather than weeks later.