Should You Block AI Training Crawlers? GPTBot, ClaudeBot and Google-Extended
Blocking an AI training crawler is a decision about whether your words train a future model. It is not, on any vendor’s documentation, a decision about whether you appear in AI answers. This guide separates the two, quotes what OpenAI, Anthropic and Google actually say, and shows the robots.txt for either choice.
By George Shvaya · · 8 min read
Three controls govern training: GPTBot from OpenAI, ClaudeBot from Anthropic, and the Google-Extended token from Google. Each vendor also operates separate agents for search indexing and for fetching a page when a person asks a question, and each documents the training control as independent of those. The whole decision turns on keeping that separation straight.
The three training controls, and what blocking each one does
Every statement in this table comes from the vendor documentation linked in the sections that follow. The last column is the one people get wrong.
| Control | Operator | What it is | Blocking it does | Blocking it does not |
|---|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train foundation models | Signals your content should not be used in training | Does not remove you from ChatGPT search (that is OAI-SearchBot) |
| ClaudeBot | Anthropic | Collects content for training; supports Crawl-delay | Signals future materials should be excluded from training datasets | Does not touch Claude-SearchBot or Claude-User, the visibility agents |
| Google-Extended | A product token, not a crawler; controls Gemini training and grounding | Opts crawled content out of training future Gemini models and grounding | Does not affect Google Search inclusion, ranking, AI Overviews or AI Mode |
OpenAI: GPTBot is training, OAI-SearchBot is visibility
OpenAI’s bot documentation says GPTBot “is used to crawl content that may be used in training our generative AI foundation models,” and that “disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” That is the entire documented consequence.
The visibility agent is a different one. OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features,” and “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.” OpenAI states that each setting is “independent of the others,” publishes GPTBot’s IP ranges at openai.com/gptbot.json, and says robots.txt changes take “~24 hours” to apply. The glossary entry on OAI-SearchBot covers the visibility side in full.
Anthropic: ClaudeBot is training, the other two are not
Anthropic’s crawler article describes ClaudeBot as the agent that collects content that “helps enhance the utility and safety of our generative AI models.” Disabling it signals that your future materials should be excluded from training datasets. Anthropic supports the non-standard Crawl-delay directive for it.
Claude-SearchBot and Claude-User are the search and user agents, and the only visibility warning in the document is attached to the latter: disabling Claude-User “may reduce visibility in user-directed web search.” No such warning is attached to ClaudeBot. Two practical notes from the same page: the bots honor robots.txt, and Anthropic discourages blocking by IP address because a bot that cannot fetch robots.txt cannot read your opt-out, which makes an IP block less reliable than the directive it replaces.
Google: a token that never appears in your logs
Google-Extended is the control most often misread, because it looks like a crawler and is not one. Google’s crawler reference says it “doesn’t have a separate HTTP request user agent string.” It is “a standalone product token” that controls whether crawled content trains future Gemini models and whether it is used for grounding in Gemini Apps and Vertex AI. Then the sentence that settles the visibility question: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”
Google’s AI features documentation adds that Google-Extended is not the control for AI Overviews or AI Mode either; those follow Googlebot and the nosnippet, max-snippet and noindex controls. We walk through the common misreading in a separate post.
Not sure what your robots.txt and firewall are actually doing?
Our free audit fetches your pages as each documented OpenAI, Anthropic and Perplexity agent, reads robots.txt per agent, and tells you which answers you are excluded from and whether that was your decision.
Get a Free AI Visibility AuditPrefer to talk it through? Book a Free Call
What Cloudflare changed, and why it matters here
On July 1, 2025, Cloudflare announced it was “changing the default to block AI crawlers unless they pay creators for their content.” The intent is a training-rights default. The effect, for a site that never opened the setting, can be broader: a block applied at the edge does not read the distinction between a training crawler and a search agent unless it is configured to. If your site sits behind Cloudflare, assume a decision has been made on your behalf and go check which agents it covers. The ten-minute method is in Is your CDN blocking AI crawlers?
A decision table by business type
None of this is data; it is reasoning from what each business has to protect and what it has to gain. Treat the lean as a starting position, not a rule.
| Business | Lean | Why |
|---|---|---|
| Publisher with original content | Block training, keep search and user agents open | The words are the product. Training use is a rights question worth deciding deliberately; visibility in answers still sends readers, so the search agents stay open. |
| Local service business | Allow everything | The site exists to be found. There is little proprietary text to protect, and any block risks catching the agents that put you in an answer when someone asks who to call. |
| SaaS company | Allow search and user agents; decide training per section | Marketing pages benefit from being cited in comparisons. Documentation may be the one place a training block is worth a conversation with legal, and it can be scoped by path. |
| Ecommerce store | Allow everything on product and category pages | Product facts in an assistant’s answer are a referral. A training block buys little and is easy to over-apply to the pages that sell. |
Whichever row you are in, the visibility groups do not change. The question is only whether the training group says Allow or Disallow.
The robots.txt for each choice
Both files below keep the search index agents and the user-initiated fetchers open. They differ in exactly one group. If you would rather generate than hand-edit, our robots.txt generator writes the same three groups, and the full agent list by job is in which AI crawlers visit your site.
If you allow training
# Visibility: search index agents. Keep open to be cited.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# Visibility: user-initiated fetchers. Keep open.
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
# Training: a separate decision. This version allows it.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlIf you block training
# Visibility: unchanged. These stay open.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
# Training: opted out. Only these three lines differ.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlGoogle-Extended is listed as a user agent in robots.txt even though it never sends requests under that name; that is how the token is set. Remember the OpenAI note that changes take about a day to apply before you test.
How to check you did not block the wrong thing
- Read robots.txt per agent, not as a whole. The group that matches OAI-SearchBot must not disallow the root. A wildcard
Disallow: /underUser-agent: *with no OAI-SearchBot group blocks it. - Fetch your homepage as each search and user agent and compare the status code to a browser request. A 403 or a challenge page is a block regardless of what robots.txt says. The AI crawler checker does both steps for any domain.
- Check the CDN dashboard for a bot-management or AI-crawler setting you did not write, and scope it to the training agents if that is your policy.
- Verify by IP list when reading logs. A request claiming to be GPTBot from outside openai.com/gptbot.json is something else.
- Re-test after 24 hours. Then put the training decision in writing next to the robots file so the next person to edit it knows which lines cost visibility and which do not.
Where this sits in the larger job: access is the first of four things every assistant needs from a site, before identity, quotable answers and corroboration. The order is set out in the AI Search Playbook.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
Not on the published documentation. OpenAI says GPTBot is used to crawl content that may be used in training its models, that OAI-SearchBot is the agent that surfaces websites in ChatGPT search, and that each setting is independent of the others. Sites opted out of OAI-SearchBot are not shown in ChatGPT search answers; sites that only block GPTBot have made a training decision.
Does blocking ClaudeBot affect whether Claude cites me?
Anthropic documents ClaudeBot as the agent that collects content for training, and documents Claude-SearchBot and Claude-User as the agents behind search and user requests. It says disabling Claude-User may reduce visibility in user-directed web search; it does not say that about ClaudeBot. Keep the search and user agents open if you want to be cited.
Does Google-Extended affect Google Search or AI Overviews?
No. Google states that Google-Extended does not impact a site’s inclusion in Google Search and is not used as a ranking signal. It is a product token, with no separate user agent, that controls Gemini training and grounding. AI Overviews and AI Mode follow Googlebot and the normal snippet controls.
What did Cloudflare change?
On July 1, 2025 Cloudflare announced it was changing the default to block AI crawlers unless they pay creators for their content. If your site is behind Cloudflare, a block may already be in place that you did not write, and it may be catching search and user agents as well as training crawlers. Check the dashboard and test each agent.
How long does a robots.txt change take to apply?
OpenAI says about 24 hours on its side. Anthropic says its bots honor robots.txt and asks that you not block by IP, because a bot that cannot fetch robots.txt cannot read your opt-out. Test after a day, not immediately.
About the work
Webvello is an AI search optimization and index engineering company founded in 2024, based in Roseville, California. We’ve delivered 50+ projects across 37+ cities, with average client ROI in the 3–5x range on properties where the technical foundation was the binding constraint. Founder George Shvaya came to search from cybersecurity and systems engineering, which is why this guide quotes vendor documentation and ends with a verification step rather than a rule of thumb.
Sources: the OpenAI, Anthropic and Google statements above are quoted from the documentation linked in each section; the Cloudflare statement is from its July 1, 2025 announcement. All re-read on October 16, 2026. Vendors change behavior; check the linked pages before relying on a row.
Want the training decision made, documented, and verified against your firewall?
Book a Free Call