Which AI Crawlers Visit Your Website, and What Does Each One Do?

AI crawlers fall into three jobs: training crawlers that collect text for future models, search index crawlers that decide whether a page can be surfaced and cited in an AI answer, and user-initiated fetchers that open a page because someone asked about it. Each vendor publishes which agent does which, and the controls are independent.

The agents, sorted by job

Every row below comes from the vendor’s own documentation, linked in the next sections. The “honors robots.txt” column records what the vendor says, not what we have observed in logs.

OpenAI: four agents, four jobs

OpenAI’s bot documentation is the clearest of the four vendors because it states the consequence of each block.

• OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features.” This is the one that matters for citations: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers,” although they can still appear “as navigational links.” A robots.txt change takes about 24 hours to propagate on OpenAI’s side.
• GPTBot “is used to crawl content that may be used in training our generative AI foundation models.” Disallowing it “indicates a site’s content should not be used in training generative AI foundation models.” OpenAI says each setting is “independent of the others,” so blocking GPTBot does not touch search inclusion.
• ChatGPT-User appears when “users ask ChatGPT or a CustomGPT a question” and the model visits a page to answer. It “is not used for crawling the web in an automatic fashion,” and because the action is user-initiated, “robots.txt rules may not apply.” It also “is not used to determine whether content may appear in Search.”
• OAI-AdsBot “only visits pages submitted as ads” on ChatGPT. Unless you buy ads there, you will not see it.

Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic’s crawler article uses the same three-way split, with one extra: it supports the non-standard Crawl-delay directive for ClaudeBot.

• ClaudeBot collects content that “helps enhance the utility and safety of our generative AI models.” That is the training crawler.
• Claude-SearchBot indexes pages to “improve the relevance and accuracy of search results.” Anthropic says disabling it prevents indexing for search and may reduce visibility. This is the citation agent on the Claude side.
• Claude-User “supports Claude AI users” by fetching a page when a person asks about it. Disabling it “may reduce visibility in user-directed web search.”

Anthropic states its bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt,” and it discourages IP blocking because a bot that cannot fetch robots.txt cannot read your opt-out. The IP list lives at claude.com/crawling/bots.json.

Perplexity: PerplexityBot and Perplexity-User

Perplexity’s bot guide documents two agents and explicitly denies a training role for either.

• PerplexityBot is “designed to surface and link websites in search results on Perplexity” and “is not used to crawl content for AI foundation models.” Allow it if you want to be cited there.
• Perplexity-User visits a page when a user asks a question and may include a link to it in the response. Because the request is user-initiated it “generally ignores robots.txt rules.”

For firewall rules, Perplexity recommends you “combine both User-Agent string matching and IP address verification,” using its published JSON lists as the source of truth. That advice applies equally to the other vendors: a user-agent string is trivially spoofed, an IP range is not.

Google: Googlebot does the work, Google-Extended is a switch

Google’s AI Overviews and AI Mode do not have their own crawler. Google’s AI features documentation says a page must be indexed and eligible to show with a snippet, that “there are no additional technical requirements,” and that “there’s also no special schema.org structured data that you need to add.” Googlebot’s robots.txt rules govern crawling for Search, including the AI features, and the snippet controls (nosnippet, data-nosnippet, max-snippet, noindex) limit what can be shown.

Google-Extended is different in kind. Google’s crawler reference says it “doesn’t have a separate HTTP request user agent string.” It is “a standalone product token” that controls whether crawled content trains future Gemini models and whether it is used for grounding in Gemini Apps and Vertex AI. And: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” We cover the confusion this causes in a separate post.

Why blocking training bots is a separate decision

The most common mistake we find in a crawler audit is a single rule written in 2023 that disallows “AI” wholesale, usually by listing GPTBot and ClaudeBot and then, as the vendors shipped new agents, extending the block to anything with a familiar prefix. The result is a site that opted out of training and, without noticing, also opted out of being cited.

The vendor documentation separates these cleanly:

• Training (GPTBot, ClaudeBot, the Google-Extended token) is about whether your words end up in a future model. Blocking it is a legitimate editorial or legal choice, and none of the vendors say it costs you visibility.
• Search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) is about whether your page can be retrieved and cited when someone asks a question today. Blocking it removes you from that engine’s answers.
• User-initiated fetches (ChatGPT-User, Claude-User, Perplexity-User) are a person reading your page through an assistant. Blocking them is close to blocking a browser, and two of the three say they may not honor the block anyway.

Write the file so the three groups are visibly separate, and the next person to edit it will understand what each line costs.

A robots.txt that keeps the decisions apart

# Search and answer agents: keep these open if you want to be cited
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# User-initiated fetchers: allow; several ignore robots.txt anyway
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /

# Training crawlers: a separate decision. Allow or Disallow on your terms.
User-agent: GPTBot
User-agent: ClaudeBot
Allow: /

# Gemini training and grounding: a token, not a crawler
User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml

Change the training block to Disallow: / if that is your policy. Leave the search and user groups alone unless you have decided you do not want to appear in that engine. If you would rather generate the file than hand-edit it, our robots.txt generator writes these groups for you.

Assume these agents read HTML, not your JavaScript

Googlebot renders pages, and Google documents how. The OpenAI, Anthropic and Perplexity bot pages make no such promise. They describe what each agent is for, how to block it and where its IP ranges are published. They do not say the agent executes scripts, waits for hydration, or reads content injected after load.

The engineering conclusion is conservative: anything that exists only after client-side JavaScript runs should be assumed invisible to the non-Google agentsuntil a vendor documents otherwise. For a business site that means prices, service lists, FAQs and location details belong in the server-rendered HTML, which is one of the reasons we build on frameworks that render on the server by default. It is also why we treat index coverage as the prerequisite for any generative engine optimization work: a page these agents cannot read cannot be cited, however good the copy is.

How to verify what is actually reaching you

1. Grep your server or CDN logs for each agent token in the table. Absence of OAI-SearchBot or PerplexityBot over a month is a signal worth investigating.
2. Check the firewall, not just robots.txt. A WAF rule or bot-management default can return a 403 to an agent your robots.txt allows. Request your own homepage with each user-agent string and confirm a 200.
3. Cross-check against the published IP lists. Traffic claiming to be GPTBot from an address outside openai.com/gptbot.json is not GPTBot.
4. Fetch a key page with scripts disabled and read what is left. That is roughly what the non-Google agents see.

The full sequence, from crawler access through entity clarity to measurement, is laid out in our AI Search Playbook. The glossary entry on AI crawlers covers the terminology.

Frequently asked questions

About the work

Webvello is an AI search optimization and index engineering company founded in 2024, based in Roseville, California. We’ve delivered 50+ projects across 37+ cities, with average client ROI in the 3–5x range on properties where the technical foundation was the binding constraint. Founder George Shvaya came to search from cybersecurity and systems engineering, which is why this article reads vendor documentation and server logs rather than repeating folklore about “AI bots.”

Sources: every agent description above is quoted or paraphrased from the vendor documentation linked in each section, all re-read on October 15, 2026. Vendors add agents and change behavior; check the linked pages before relying on a row.