Automate Search-Engine URL Submission: A Python IndexNow + Google Indexing API Pipeline
You shipped 400 new programmatic pages last night. Or you rewrote 30 title tags. Or you fixed a canonical that was bleeding equity for months. None of it matters until a search engine recrawls the URLs — and left to passive discovery, that can take days for a healthy site and weeks for a large or low-authority one. Every day a changed URL sits unrecralled is a day you rank on stale content.
Most teams treat indexing as something that “just happens.” The teams that win at scale treat it as a pipeline: the moment a URL is published or changed, a job fires that tells the search engines about it. This post is the working version of that pipeline in Python — built on IndexNow and the Google Indexing API — plus an honest explanation of what each channel can and cannot do, because that part is where most tutorials quietly mislead you.
What each submission channel actually does
Before any code, get the constraints straight. Conflating these is the single most common mistake in indexing automation.
IndexNow is an open protocol supported by Microsoft Bing, Yandex, Seznam, and Naver. You ping one endpoint, participating engines share the signal. It accepts any URL on a verified host, in batches of up to 10,000 per request. It is the broad, no-strings channel — and notably, Google is not a participant.
Google Indexing API is narrower than most SEOs admit. Google officially supports it only for pages with JobPosting or BroadcastEvent structured data. Submitting arbitrary URLs through it is off-label: it may trigger a crawl, but Google does not guarantee or commit to indexing, and leaning on it for general content is building on sand.
Google, for everything else, still relies on your XML sitemap (with accurate lastmod dates) and natural crawling. The reliable automated nudge is to keep lastmod truthful and resubmit the sitemap, then verify recrawl with the Search Console URL Inspection API rather than assume it happened.
So the honest architecture is: IndexNow for broad cross-engine coverage, the Indexing API for your genuinely supported page types, and sitemap freshness + inspection for Google’s general content. Anyone selling you “instant Google indexing for any URL” is selling you a quota you’ll burn for nothing.
The pipeline at a glance
The design is event-driven with a safety net. On publish or update, your CMS (or a webhook, or an programmatic SEO pipeline with a quality gate before indexing) drops the changed URL onto a queue. A worker deduplicates, then fans the URL out to the appropriate channels. A nightly reconciler re-checks anything that hasn’t shown up in coverage. Let’s build the two submission clients, then wire them together.
Step 1 — Stand up your IndexNow key
IndexNow proves you own the host with a key file. Generate a key (a UUID works), and host it as a plain-text file at the site root whose filename is the key itself.
python -c "import uuid; print(uuid.uuid4().hex)"
# -> 5b2c1f9e8a734d6c9f01a2b3c4d5e6f7
# Upload the file so this resolves with the key as its body:
# https://example.com/5b2c1f9e8a734d6c9f01a2b3c4d5e6f7.txt
That’s the entire verification step. No OAuth, no dashboard.
Step 2 — Submit URLs to IndexNow in bulk
One POST to api.indexnow.com covers every participating engine. Batch your URLs; never loop one request per URL.
import requests
INDEXNOW_KEY = "5b2c1f9e8a734d6c9f01a2b3c4d5e6f7"
HOST = "example.com"
def submit_indexnow(urls: list[str]) -> int:
payload = {
"host": HOST,
"key": INDEXNOW_KEY,
"keyLocation": f"https://{HOST}/{INDEXNOW_KEY}.txt",
"urlList": urls[:10000], # hard cap per request
}
r = requests.post(
"https://api.indexnow.com/indexnow",
json=payload,
headers={"Content-Type": "application/json; charset=utf-8"},
timeout=30,
)
# 200 = accepted, 202 = accepted/pending, 422 = URLs don't match host,
# 403 = key not valid, 429 = too many requests
return r.status_code
print(submit_indexnow([
"https://example.com/guides/widget-sizing",
"https://example.com/guides/widget-pricing",
]))
Two response codes deserve attention. A 422 almost always means a URL’s host doesn’t match the verified host (a stray www or http), and a 429 means slow down — batch larger and submit less often rather than retrying in a tight loop.
Step 3 — Google Indexing API for supported page types
Use this only for JobPosting/BroadcastEvent URLs. Create a Google Cloud service account, enable the Indexing API, and add the service account’s email as an owner of the property in Search Console — without that ownership grant every call returns 403 Permission denied.
from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession
SCOPES = ["https://www.googleapis.com/auth/indexing"]
ENDPOINT = "https://indexing.googleapis.com/v3/urlNotifications:publish"
creds = service_account.Credentials.from_service_account_file(
"service-account.json", scopes=SCOPES
)
session = AuthorizedSession(creds)
def notify_google(url: str, action: str = "URL_UPDATED") -> dict:
# action is "URL_UPDATED" on publish/edit, "URL_DELETED" on removal
resp = session.post(ENDPOINT, json={"url": url, "type": action}, timeout=30)
resp.raise_for_status()
return resp.json()
print(notify_google("https://example.com/jobs/senior-python-engineer"))
The default quota is 200 requests per day. You can batch up to 100 notifications into a single HTTP call via the batch endpoint, which is worth implementing the moment you have more than a handful of job or event pages turning over daily.
Step 4 — Nudge Googlebot for ordinary content, then verify
For standard articles and landing pages, the Indexing API is the wrong tool. Keep your sitemap’s lastmod honest on every change, then confirm Google actually came back using the URL Inspection API rather than assuming. This is the same machinery covered in depth in monitoring index coverage with the URL Inspection API — the submission pipeline and the verification pipeline are two halves of one loop.
from googleapiclient.discovery import build
# creds scoped to webmasters.readonly
sc = build("searchconsole", "v1", credentials=gsc_creds)
def last_crawl_state(url: str, site: str) -> dict:
body = {"inspectionUrl": url, "siteUrl": site}
res = sc.urlInspection().index().inspect(body=body).execute()
idx = res["inspectionResult"]["indexStatusResult"]
return {
"coverage": idx.get("coverageState"),
"last_crawl": idx.get("lastCrawlTime"),
"verdict": idx.get("verdict"),
}
If a URL submitted three days ago still shows coverageState of “Discovered – currently not indexed,” that is a quality or crawl-budget signal, not a submission problem — throwing more pings at it won’t help, and may waste quota.
Step 5 — Wire it into your publish flow
Glue the clients together behind one function, deduplicate so a URL edited five times in an hour is submitted once, and route by page type.
def dispatch(url: str, page_type: str):
submit_indexnow([url]) # always: broad coverage
if page_type in ("job_posting", "broadcast_event"):
notify_google(url) # only supported types
# ordinary pages rely on sitemap lastmod + later verification
# Called from a webhook on publish/update, with a Redis set or DB
# unique constraint upstream to collapse duplicates within the window.
Run the dispatcher off your CMS publish webhook, and run a nightly reconciler that pulls the day’s changed URLs, checks coverage with the inspection client, and re-queues only the ones that genuinely haven’t been recrawled. If you already orchestrate workflows visually, the same logic drops cleanly into the agentic patterns we cover when exposing Search Console to an AI agent through an MCP server.
What to actually expect
Set expectations honestly, because this is where credibility is won or lost. On the Bing/IndexNow side, recrawl of submitted URLs typically drops from days to hours, and it’s free and effectively unlimited at sane volumes — a clear win, especially as Bing’s index increasingly feeds AI answer engines. On the Google side, submission shortens the discovery step but never overrides Google’s quality judgement: a thin page submitted instantly is still a thin page. The realistic payoff is a tighter feedback loop — you change something and learn within hours whether Google recrawled, instead of guessing for a week. For large or programmatic sites, that compounds fast.
The trap to avoid is treating submission as a ranking lever. It is a latency lever. It removes the wait, not the work.
Key takeaways
Automate submission at the moment of change, not on a slow cron. Use IndexNow for broad, free, cross-engine coverage; reserve the Google Indexing API for its officially supported JobPosting and BroadcastEvent pages; and for everything else, keep sitemaps fresh and verify recrawl with the URL Inspection API instead of assuming. Deduplicate aggressively, respect the quotas, and remember that fast indexing of low-quality pages just gets your problems discovered faster.
If you found this useful, bookmark SEO Automation Club and check back — we publish working automation playbooks (with real code, not listicles) every week. And if indexing is your bottleneck, the natural next read is our deep dive on automating index-coverage monitoring with Python, which closes the verification half of this loop.
Frequently asked questions
Does IndexNow help with Google rankings?
Not directly — Google does not participate in IndexNow. It accelerates discovery on Bing, Yandex, Seznam, and Naver. Given Bing increasingly powers AI answer engines, that coverage matters more than it used to, but it is a discovery accelerant, not a Google ranking factor.
Can I submit any URL to the Google Indexing API?
Technically the endpoint accepts any URL, but Google officially supports it only for pages with JobPosting or BroadcastEvent structured data. Using it for general content is unsupported, may do nothing, and burns your daily quota — rely on sitemap freshness and the URL Inspection API for ordinary pages instead.
What are the rate limits I need to respect?
IndexNow accepts up to 10,000 URLs per request and is generous on volume, though a 429 means batch larger and submit less frequently. The Google Indexing API defaults to 200 requests per day, with up to 100 notifications per batched HTTP call.
How do I confirm a submission actually worked?
Don’t trust the submission response alone. Query the Search Console URL Inspection API for the URL and read coverageState and lastCrawlTime. If a URL stays “Discovered – currently not indexed” after several days, treat it as a quality or crawl-budget issue rather than resubmitting.
