Automate Broken Link and 404 Monitoring: A Python + GSC Pipeline That Suggests Redirects

Most teams treat broken links as a one-time cleanup job. You run a crawl before a migration, map the dead URLs to redirects, ship it, and move on. But link rot is not an event — it is a steady drip. External pages you reference get deleted, internal templates change a slug, a product goes out of stock and its page 410s, an editor fat-fingers an anchor. Every one of those is a small leak: wasted crawl budget, dead-ended link equity, and a user who hit a 404 instead of the thing they wanted.

This guide builds a continuous broken-link and 404 monitoring pipeline in Python. It is deliberately different from a migration redirect map (a one-shot batch job). This runs on a schedule, reconciles three independent signals, classifies what it finds, and — the part most tutorials skip — suggests a redirect target using embeddings so a human only reviews the edge cases. Everything here is built from free or freemium tools: httpx, the Google Search Console API, and a self-hosted n8n or cron schedule.

Why broken-link monitoring deserves its own pipeline

A redirect map answers the question “where should these specific old URLs point?” once. A monitor answers a different, ongoing question: “what is broken right now, and is it getting worse?” The two failure modes you are hunting are not the same either:

• Internal broken links — a link on your own site pointing to a URL that 404s. These are pure self-inflicted crawl-budget waste and you should fix the link at the source, not just redirect the target.
• External broken links — you link out to a page that died. The fix is to update or remove the reference, which is an editorial action, not a server rule.
• Soft 404s — a page that returns 200 OK but is functionally empty (“no results found”, an out-of-stock shell). Google treats these as 404s anyway, and a status-code-only checker will miss them entirely.
• Redirect chains and loops — a link that technically resolves but takes three hops to get there, leaking a little equity and latency at each step.

A pipeline that only checks HTTP status codes catches the first two and silently ignores the rest. That is the gap we are closing.

Architecture: four stages, three signals

The design reconciles three independent sources of truth so you are not blindly trusting any single one:

1. Your own crawl — the only source that sees where a broken link lives (which page it is on, what the anchor text is).
2. Google Search Console — the only source that tells you what Googlebot actually experienced and indexed.
3. Server logs — the authoritative record of every real 404 served, including ones no crawler would ever find because they come from old external backlinks.

The stages: (1) crawl and check status, (2) reconcile with GSC and logs, (3) classify, (4) suggest a redirect and route to a review queue or alert.

Step 1 — Crawl internal links and check status (async)

Start with an async crawler. The point is concurrency: checking 20,000 links serially over the network is the difference between a 4-minute run and a 2-hour one. httpx with an asyncio semaphore keeps you fast without hammering your own origin.

import asyncio, httpx
from urllib.parse import urljoin, urlparse
from selectolax.parser import HTMLParser

SEM = asyncio.Semaphore(20)  # cap concurrency; be polite to your own server
SITE = "https://example.com"

async def fetch(client, url):
    async with SEM:
        try:
            r = await client.get(url, follow_redirects=True, timeout=15)
            return url, r.status_code, str(r.url), len(r.history)
        except httpx.HTTPError as e:
            return url, None, None, 0

async def extract_links(client, page):
    r = await client.get(page, timeout=15)
    tree = HTMLParser(r.text)
    out = []
    for a in tree.css("a[href]"):
        href = urljoin(page, a.attributes.get("href", ""))
        if urlparse(href).scheme in ("http", "https"):
            out.append((page, href, (a.text() or "").strip()))
    return out

The two things people forget: follow redirects so you can measure chain length (len(r.history)), and capture the source page and anchor text for every link. Those two fields are what make the redirect suggestion in Step 4 possible — you cannot recommend a target if you do not know what the link was trying to say.

Step 2 — Reconcile with Search Console and logs

Here is a detail that trips up a lot of automation: Google retired the standalone “Crawl Errors” report years ago, and there is no clean API that hands you a list of 404s. The practical sources are the URL Inspection API (one URL at a time, rate-limited — good for confirming a suspect, useless for discovery) and the bulk data export to BigQuery, where the Index Coverage states (including Not found (404) and Soft 404) land as queryable rows.

# Pull 404 / soft-404 URLs from the GSC bulk export in BigQuery
from google.cloud import bigquery
client = bigquery.Client()
q = """
SELECT url, type
FROM `project.searchconsole.url_coverage`
WHERE type IN ('Not found (404)', 'Soft 404')
  AND data_date = CURRENT_DATE() - 1
"""
gsc_404 = {row["url"]: row["type"] for row in client.query(q)}

If you have not set up the bulk export, your server logs are the faster path to authoritative 404 data, and they catch the long tail of dead inbound backlinks that no crawl will surface. The same parsing approach used for crawl-budget work applies directly here — group by status code, filter to 404 and 410, and join on the URL.

Step 3 — Classify before you act

Never pipe raw broken URLs straight into a redirect rule. Classify first, because each class has a different correct action:

def classify(row):
    if row["status"] in (404, 410) and row["is_internal"]:
        return "internal_broken"      # fix the link at source
    if row["status"] in (404, 410):
        return "external_broken"      # editorial: update or remove
    if row["status"] == 200 and row["thin"]:
        return "soft_404"             # investigate template / inventory
    if row["hops"] >= 2:
        return "redirect_chain"       # collapse to a single hop
    return "ok"

The thin flag for soft-404 detection is a small heuristic — word count below a floor, or the presence of strings like “no results” / “out of stock” in the main content. It is crude, but it surfaces the exact pages a status-code check would wave through. This classification step is the same discipline behind automating index coverage monitoring — you reconcile signals and bucket by required action rather than reacting to a single metric.

Step 4 — Suggest a redirect with embeddings

This is where the pipeline earns its keep. For every external_broken or genuinely-retired internal URL that needs a redirect, you want to propose the most semantically similar live URL instead of dumping it on someone’s desk. Embed your live URL inventory once, embed the dead URL’s context (its slug plus the anchor text people used to link to it), and take the nearest neighbour by cosine similarity.

import numpy as np
from openai import OpenAI
oai = OpenAI()

def embed(texts):
    r = oai.embeddings.create(model="text-embedding-3-small", input=texts)
    return np.array([d.embedding for d in r.data])

live_urls   = [...]                 # your current, indexable URL inventory
live_vecs   = embed(live_urls)

def suggest(dead_url, anchor_texts):
    ctx = dead_url.split("/")[-1].replace("-", " ") + " " + " ".join(anchor_texts)
    v = embed([ctx])[0]
    sims = live_vecs @ v / (np.linalg.norm(live_vecs, axis=1) * np.linalg.norm(v))
    i = int(sims.argmax())
    return live_urls[i], float(sims[i])

target, score = suggest("https://example.com/old-guide", ["python seo guide"])
# score >= 0.75 -> auto-queue the redirect; below -> route to a human

Set a confidence threshold (0.75 is a sane starting point) and only auto-stage redirects above it. Everything below goes to a human queue with the top three candidates attached, so even the “manual” cases are a one-click decision. This is the same embedding-similarity engine behind embedding-based 301 redirect mapping; here we are applying it incrementally instead of in one migration batch.

Step 5 — Alert, queue, and gate

The output is two streams. High-confidence, low-risk fixes (internal broken links, redirect chains, redirects above threshold) go into a staged change set you can review in bulk — a Google Sheet or a pull request against your redirects file. Anything novel or low-confidence fires a Slack alert with the source page, anchor text, and suggested targets. Crucially, nothing auto-applies to production without a gate. A confidently-wrong redirect is worse than a 404 because it is invisible to your users and harder to catch later. If you crawl with a headless renderer to catch JavaScript-injected links, the trade-offs in this automated crawler teardown are worth reading before you pick a tool.

What to expect once it’s running

On a mid-size content site (~8,000 indexable URLs) a weekly run typically surfaces a handful of internal broken links introduced by routine editing, a slower trickle of external rot, and — the surprise for most teams — a pile of soft 404s on thin tag/filter pages that were never flagged because they return 200. The embedding suggester comfortably auto-resolves the majority of clear-cut external redirects above the 0.75 threshold, which collapses what used to be an afternoon of manual mapping into a five-minute review. The real win is not any single fix; it is that link rot stops accumulating silently between migrations.

Start small: run the crawler against your top templates, eyeball the classifications for a week, then turn on the suggester in shadow mode (log proposals, apply nothing) until you trust the threshold. Only then wire it to your redirects file behind a review gate.

Frequently asked questions

How is this different from a migration redirect map?

A migration map is a one-time batch that moves a known set of old URLs to new ones. This is a recurring monitor that detects newly broken links and soft 404s as they appear, and proposes targets incrementally. They share the embedding-similarity engine but run on completely different cadences and triggers.

Can I get a clean list of 404s straight from the Google Search Console API?

Not directly. Google retired the standalone Crawl Errors report and there is no endpoint that returns a 404 list. Use the URL Inspection API to confirm individual suspects, the bulk data export to BigQuery to query Index Coverage states at scale, and your server logs as the authoritative record of every 404 actually served.

Should broken internal links be redirected or fixed at the source?

Fixed at the source. A redirect on an internal link still forces an extra hop and leaves the wrong URL embedded in your own templates. Reserve redirects for URLs you genuinely cannot edit the links to — mostly external backlinks pointing at pages you retired.

Is it safe to auto-apply the embedding-suggested redirects?

Only above a confidence threshold and behind a review gate. A confidently-wrong redirect sends users and crawlers somewhere irrelevant and is harder to detect than an honest 404. Run the suggester in shadow mode first, calibrate the threshold against your own decisions, and keep a human in the loop for anything below it.

Found this useful? Bookmark SEO Automation Club and subscribe for a new working automation playbook every week — real code, real workflows, no fluff. If you are building out your technical monitoring stack, the companion guide on automating index coverage monitoring pairs naturally with this pipeline.