Build an Index Coverage Watchdog: Automate the GSC URL Inspection API to Catch Silent Deindexing

Google will happily drop your pages from the index without telling you. There is no email, no red banner in Search Console — just a slow bleed of impressions that you notice three weeks later, when the traffic report finally looks wrong. The aggregate Page Indexing report in GSC is too coarse to catch it early: it updates on a lag, groups URLs into buckets, and won’t tell you which money page silently flipped from “indexed” to “Crawled – currently not indexed” last Tuesday.

The URL Inspection API gives you per-URL, per-day index state straight from Google — the same verdict you’d get by pasting a URL into the inspection bar, but scriptable. In this post we’ll build an index coverage watchdog: a Python pipeline that inspects your URLs on a tiered schedule, stores every verdict in SQLite, diffs runs to detect dangerous state transitions, and pings Slack the day a page falls out — not the month after.

The constraint that shapes everything: 2,000 inspections per day

The URL Inspection API is generous in data and stingy in volume. Each verified Search Console property gets roughly 2,000 inspections per day and 600 queries per minute. That single number dictates the architecture. If your site has 800 URLs, you can inspect everything daily and skip the clever parts. If it has 50,000, you cannot — a full sweep would take 25 days, which defeats the purpose of a watchdog.

The answer is a tiered priority queue. Here is the budget math for a hypothetical 30,000-URL site, allocating the daily 2,000-call budget:

A worked quota budget for a 30,000-URL site

• Tier 1 — money pages (300 URLs): inspected every day. Product pages, top commercial landers, anything that pays rent. Cost: 300 calls/day.
• Tier 2 — traffic pages (4,900 URLs): every page with a click in GSC in the last 28 days. Inspected weekly, so ~700 calls/day.
• Tier 3 — the long tail (24,800 URLs): inspected monthly, ~830 calls/day.
• Reserve (~170 calls/day): for on-demand checks — newly published URLs, post-deploy verification, anything an alert flags for re-inspection tomorrow.

With this split, a deindexing event on a page that matters is caught within 24 hours; on a mid-tier page, within a week. That’s the trade the quota forces, and it’s a good one: the pages most likely to hurt when they vanish are also the ones checked most often. Tier membership should be recomputed weekly from GSC click data, not hand-maintained — pages migrate between tiers as their traffic changes.

The verdict taxonomy: which coverage states actually matter

The API returns a coverageState string per URL. Most tutorials list them; few tell you which transitions to alarm on. After you diff a few weeks of runs, a clear hierarchy emerges:

Page-one alerts (wake someone up)

• Submitted and indexed → Crawled - currently not indexed: the classic silent deindex. Google recrawled the page and chose to drop it. On a Tier 1 URL this is a fire.
• Anything → Excluded by 'noindex' tag: almost always a deploy accident, not a Google decision. Your fix is in your codebase, and every day it ships costs you.
• Anything → Duplicate, Google chose different canonical than user: Google is now consolidating your page into another URL — check googleCanonical vs userCanonical in the response to see exactly where the equity went.

Watch-list transitions (log, don’t page)

• Discovered - currently not indexed lingering more than ~30 days on a Tier 2 page: a crawl-budget or perceived-quality signal, best handled by strengthening internal links rather than resubmitting.
• Page with redirect appearing on a URL you didn’t redirect: usually a trailing-slash or http/https inconsistency introduced upstream.

One non-obvious field worth storing: lastCrawlTime. A Tier 1 page whose last crawl date stops advancing for 45+ days is drifting out of Google’s attention even if its coverage state still reads “indexed” — that’s an early-warning signal no coverage state will give you.

Building the watchdog in Python

You need a Google Cloud service account with the Search Console API enabled, added as a (full or restricted) user on the property. Then the core inspect call is small:

from google.oauth2 import service_account
from googleapiclient.discovery import build

SCOPES = ["https://www.googleapis.com/auth/webmasters.readonly"]
creds = service_account.Credentials.from_service_account_file(
    "sa.json", scopes=SCOPES)
svc = build("searchconsole", "v1", credentials=creds)

def inspect(url: str, prop: str) -> dict:
    body = {"inspectionUrl": url, "siteUrl": prop}
    resp = svc.urlInspection().index().inspect(body=body).execute()
    r = resp["inspectionResult"]["indexStatusResult"]
    return {
        "url": url,
        "coverage": r.get("coverageState"),
        "verdict": r.get("verdict"),
        "google_canonical": r.get("googleCanonical"),
        "user_canonical": r.get("userCanonical"),
        "last_crawl": r.get("lastCrawlTime"),
        "robots": r.get("robotsTxtState"),
    }

The state store is a single SQLite table keyed on (url, run_date). The interesting part is the differ, which compares each URL’s latest verdict against its previous one and classifies the transition:

ALERT_TO = {"Crawled - currently not indexed",
            "Excluded by 'noindex' tag",
            "Duplicate, Google chose different canonical than user"}

def classify(prev: str | None, curr: str) -> str:
    if prev == curr:
        return "stable"
    if curr in ALERT_TO and (prev or "").startswith("Submitted"):
        return "ALERT"
    if prev is None:
        return "new"
    return "watch"

Wrap the daily run in a scheduler that drains each tier’s due-list in order, sleeps to respect the per-minute limit, and stops cleanly at the daily budget. When an ALERT fires, two automatic follow-ups earn their keep: tomorrow’s run re-inspects the URL from the reserve budget (one bad verdict can be a transient), and a confirmed noindex regression cross-checks against your own crawl — if your server is actually serving the noindex, the bug is yours, and the alert should say so. We covered that guardrail crawl in depth in our robots.txt and meta-robots monitoring pipeline; the two systems are natural partners — one watches what you serve, the other watches what Google concluded.

What to do with a confirmed drop

Detection without a playbook is just faster anxiety. A reasonable response ladder, in order of cost:

• Noindex or robots regression: revert the deploy, then request recrawl. This is the only case where immediate resubmission clearly helps — our IndexNow + Indexing API submission pipeline can be triggered directly from the alert handler.
• “Crawled – currently not indexed” on real content: resubmission alone rarely sticks. Treat it as a quality-and-links problem: add 2–3 contextual internal links from strong pages, refresh thin sections, then resubmit once.
• Canonical hijacked to another of your URLs: decide which page you actually want ranking, consolidate the loser (301 or canonical), and stop splitting signals.

The watchdog’s SQLite history also makes post-incident analysis honest: you can answer “when did this start, and what did we deploy that week?” with timestamps instead of memory. If you’d rather have an agent do that correlation step for you, this dataset is exactly the kind of thing we handed to Claude Code in our agentic technical-SEO triage experiment — a coverage-state timeline plus a deploy log is a triage prompt that practically writes itself.

Pitfalls that will bite you

• The API reflects the last crawl, not live reality. If you fixed a noindex an hour ago, inspection still shows the old state until Google recrawls. Pair every “is it fixed?” check with a live fetch of your own page.
• Property mismatch fails loudly, quota waste fails silently. The siteUrl must be the exact verified property (domain property syntax is sc-domain:example.com). And inspecting URLs that redirect wastes budget on verdicts about the wrong URL — canonicalize your inspection list first.
• Don’t alert on single flips for Tier 3. Long-tail pages oscillate in and out of the index routinely. Require two consecutive bad verdicts before alerting below Tier 1, or your Slack channel becomes noise and the one alert that matters gets muted.
• Quota is per property, not per account. Large sites can be verified as multiple valid properties (subfolder properties) to multiply effective budget — legitimate, documented, and the difference between weekly and daily coverage for a six-figure-URL site.

FAQ

How many URLs can I check with the URL Inspection API per day?

Around 2,000 per day per verified Search Console property, with a burst limit of 600 per minute. The budget resets daily, and verifying subfolders as separate properties multiplies your effective quota.

Does inspecting a URL trigger Google to recrawl it?

No. Inspection is read-only — it reports the state from Google’s last crawl. To request recrawling you still need manual “Request Indexing” in the UI or a submission pipeline (IndexNow, sitemaps, internal links).

How often should I run index coverage checks?

Daily for your commercially critical pages, weekly for pages with recent GSC clicks, monthly for the long tail. Tier membership should be recomputed automatically from Search Console click data, not maintained by hand.

Why does the Page Indexing report in GSC not show a page I know was deindexed?

The aggregate report updates on a multi-day lag and samples URL buckets. The URL Inspection API is per-URL and current as of Google’s last crawl, which is why it catches individual drops days or weeks earlier.