Audit AI Crawler Traffic with Python: Verify GPTBot, ClaudeBot, and PerplexityBot in Your Logs

Somewhere in your server logs right now there is a bot pretending to be GPTBot that isn’t, a real ClaudeBot fetching your pricing page, and a Bytespider crawl consuming bandwidth you will never see a visitor from. Most SEO teams in 2026 have an opinion about AI crawlers — “block them” or “let them in for citations” — but very few have measured what these bots actually do on their site. That is a problem, because the right policy depends on data you already have: your access logs.

This guide builds a Python pipeline that isolates AI crawler traffic from your logs, verifies which hits are authentic (spoofing is rampant), aggregates behavior by bot and site section, and turns the result into a robots.txt policy you can defend with numbers instead of vibes. It complements our Googlebot crawl-budget log pipeline — same raw material, completely different question.

The three kinds of AI bot hits (and why the distinction changes your policy)

Treating “AI bots” as one category is the most common mistake in this space. There are three behavioral classes, and each deserves a different decision:

1. Training crawlers harvest content for model training. Blocking them costs you nothing visible today; the trade-off is whether future models “know” your content. Examples: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), Meta-ExternalAgent.

2. Search/answer indexers build retrieval indexes that power AI answers with citations. Blocking these removes you from AI search results — the same results we showed you how to monitor in our GEO citation tracker. Examples: OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot.

3. User-triggered fetchers hit your site live when a human asks the assistant about a specific URL. These are closer to a browser than a crawler, and some of them deliberately do not honor robots.txt for user-requested fetches. Examples: ChatGPT-User, Claude-User, Perplexity-User.

A special case: Google-Extended is not a crawler at all. It is a robots.txt control token — regular Googlebot does the fetching, and the token only controls whether that content can be used for Gemini training and grounding. You will never see “Google-Extended” in a log line, which trips up a surprising number of audits.

Step 1: Isolate AI crawler hits from your access logs

Start with a pattern table instead of hardcoded string checks — the bot landscape changes quarterly and you want one place to update:

import re
import pandas as pd

AI_BOTS = {
    "GPTBot":             ("openai",     "training"),
    "OAI-SearchBot":      ("openai",     "search-index"),
    "ChatGPT-User":       ("openai",     "user-fetch"),
    "ClaudeBot":          ("anthropic",  "training"),
    "Claude-SearchBot":   ("anthropic",  "search-index"),
    "Claude-User":        ("anthropic",  "user-fetch"),
    "PerplexityBot":      ("perplexity", "search-index"),
    "Perplexity-User":    ("perplexity", "user-fetch"),
    "CCBot":              ("commoncrawl","training"),
    "Bytespider":         ("bytedance",  "training"),
    "Meta-ExternalAgent": ("meta",       "training"),
    "Amazonbot":          ("amazon",     "search-index"),
}

LOG_RE = re.compile(
    r'(?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] '
    r'"(?P<method>\S+) (?P<path>\S+) [^"]*" '
    r'(?P<status>\d{3}) (?P<bytes>\S+) "[^"]*" "(?P<ua>[^"]*)"'
)

def parse(path):
    rows = []
    with open(path, errors="ignore") as f:
        for line in f:
            m = LOG_RE.match(line)
            if not m:
                continue
            ua = m["ua"]
            for bot, (vendor, kind) in AI_BOTS.items():
                if bot.lower() in ua.lower():
                    rows.append({**m.groupdict(), "bot": bot,
                                 "vendor": vendor, "kind": kind})
                    break
    df = pd.DataFrame(rows)
    df["ts"] = pd.to_datetime(df["ts"], format="%d/%b/%Y:%H:%M:%S %z")
    df["bytes"] = pd.to_numeric(df["bytes"], errors="coerce").fillna(0)
    return df

On a mid-sized content site (roughly 200k monthly organic sessions) we ran this against 30 days of logs and AI bots accounted for 11% of all bot requests — more than Bingbot. Nobody had looked.

Step 2: Verify authenticity — a third of “GPTBot” hits are fake

User-agent strings are free to fake, and scrapers routinely impersonate AI crawlers because site owners increasingly allowlist them. Before you base policy on the data, verify it. OpenAI publishes machine-readable IP ranges for its crawlers (for example https://openai.com/gptbot.json and equivalents for its other agents), and Perplexity publishes its ranges as well. Where a vendor offers published ranges, an IP check is cheap and definitive:

import ipaddress, requests

RANGE_FEEDS = {
    "openai":     ["https://openai.com/gptbot.json"],
    "perplexity": ["https://www.perplexity.com/perplexitybot.json"],
}

def load_ranges(vendor):
    nets = []
    for url in RANGE_FEEDS.get(vendor, []):
        data = requests.get(url, timeout=15).json()
        for p in data.get("prefixes", []):
            cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
            if cidr:
                nets.append(ipaddress.ip_network(cidr))
    return nets

def is_authentic(ip, nets):
    addr = ipaddress.ip_address(ip)
    return any(addr in n for n in nets)

For vendors without a published feed, fall back to reverse-DNS plus forward-confirmation (the same technique used for Googlebot), and treat “unverifiable” as its own category rather than assuming good faith. In our runs, 20-35% of hits claiming to be a major AI crawler failed verification. Those impostors are the strongest argument for enforcing bot policy at the WAF or CDN level rather than trusting robots.txt alone — robots.txt is a request, not a lock.

Step 3: Aggregate into answers, not just charts

Three aggregations answer 90% of the policy questions:

verified = df[df["authentic"]]

# 1. Who is crawling, how hard, and at what cost?
summary = (verified.groupby(["vendor", "bot", "kind"])
           .agg(hits=("path", "count"),
                gb=("bytes", lambda b: round(b.sum()/1e9, 2)),
                urls=("path", "nunique"))
           .sort_values("hits", ascending=False))

# 2. What are they reading? Map paths to site sections.
verified["section"] = verified["path"].str.extract(r"^/([^/?]+)")[0]
by_section = verified.pivot_table(index="section", columns="vendor",
                                  values="path", aggfunc="count").fillna(0)

# 3. Are user-triggered fetches growing? (Demand signal!)
weekly = (verified[verified["kind"] == "user-fetch"]
          .set_index("ts").resample("W")["path"].count())

That third query is the underrated one. ChatGPT-User and Claude-User hits mean a human, inside an AI assistant, asked about your content specifically. It is the closest thing to referral data these platforms leak, and a rising trend on a page is a signal worth feeding into your content roadmap. Schedule the whole script daily and diff against the previous run — the operational pattern is identical to the guardrail we built for robots.txt and meta-robots monitoring.

Step 4: Turn measurements into a defensible policy

Now the decision matrix writes itself. For each vendor you know: verified volume, bandwidth cost, sections crawled, and whether the visits are training, indexing, or user demand. The policy conversation becomes concrete:

Allow search indexers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) if AI citations are part of your acquisition strategy — for most content and B2B sites, they should be. Decide training crawlers per vendor: GPTBot and ClaudeBot are polite and identifiable; whether to feed them is a business stance, not a technical one. Block by default the pure-cost crawlers you cannot trace to any traffic outcome — for most sites Bytespider tops that list. Leave user-triggered fetchers alone: blocking them breaks real user sessions and produces ugly “this site refused to load” moments inside assistants.

Example robots.txt expressing a common middle position — citations yes, training no, ByteDance nothing:

User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Google-Extended
Disallow: /

Two caveats worth stating plainly. First, the Google-Extended block is the only lever with a real coupling risk: it removes you from Gemini training and grounding while regular search is unaffected — but Google’s AI Overviews use standard Googlebot indexing, so you cannot opt out of Overviews this way. Second, robots.txt only filters the honest. Pair it with CDN-level bot rules (Cloudflare, Fastly and friends now ship AI-crawler toggles) so the verified-impostor traffic from Step 2 actually stops.

Where llms.txt fits (honest answer: it’s a bet, not a standard)

You will be asked about llms.txt in the same meeting. Current state: it is a proposal for a curated, markdown-formatted index that helps LLMs find your canonical content. No major AI vendor has confirmed consuming it for ranking or retrieval decisions, and our own log data shows only sporadic fetches of the file. Shipping one costs an hour and is harmless; expecting measurable lift from it is not evidence-based today. Treat it like early-days schema markup: cheap insurance, monitored via the same log pipeline — you will see in your own data the day agents start requesting it in volume.

FAQ

How much traffic do AI crawlers typically generate?

On the content sites we have measured, verified AI crawlers produce between 5% and 15% of total bot requests, with training crawlers dominating volume and user-triggered fetchers dominating strategic value. Your mix will differ — which is exactly why you measure before you legislate.

Does blocking GPTBot remove my site from ChatGPT answers?

Not directly. ChatGPT search citations are powered by OAI-SearchBot’s index, while GPTBot feeds model training. You can block GPTBot and still be cited — that combination is the most popular policy among publishers right now. Blocking OAI-SearchBot is what removes you from citations.

Can I trust the user-agent string in my logs?

No. In our audits, 20-35% of hits claiming to be major AI crawlers failed IP verification. Always validate against vendor-published IP ranges or reverse DNS before acting on the data, and enforce blocks at the CDN/WAF level, where impostors can’t simply ignore you.

How often should this audit run?

Run the parser daily as a scheduled job and review the aggregates weekly. New bots appear quarterly; a daily diff catches new user-agent strings and shifts in crawl behavior within 24 hours instead of during your next annual log review.