Original research · Data you can download
Who Blocks AI Crawlers? We Read the robots.txt of the Top 1,000 Websites
About one in four of the world’s top websites tells at least one AI training crawler to stay out. Only one in seven blocks an AI search crawler. The biggest sites have settled on a split: keep the training bots out, let the search bots in. We read 516 robots.txt files on November 13, 2026, to measure it.
By George Shvaya · · Updated · 13 min read
- block at least one AI training crawler
- 26.4%
- 136 of 516
- block at least one AI search crawler
- 14.5%
- 75 of 516
- block training but not search, vs the reverse
- 62 vs 1
- sites
- publish an llms.txt file
- 22.9%
- 134 of 585
Training crawlers are blocked twice as often as search crawlers
This section answers which AI agents the top sites turn away. Every AI company now runs more than one agent, and they do different jobs. A training crawler collects pages to build future models. A search crawler builds the index an assistant searches when it answers with sources. A user agent fetches one page because a person asked the assistant about it. OpenAI, Anthropic and Perplexity document each one separately, and a site can allow one and block another (OpenAI, Anthropic, Perplexity).
| Agent | Job | Blocked | Share | Named in the file |
|---|---|---|---|---|
| Bytespider (ByteDance) | Model training | 113 | 21.9% | 109 |
| CCBot (Common Crawl) | Model training | 109 | 21.1% | 115 |
| GPTBot (OpenAI) | Model training | 97 | 18.8% | 134 |
| ClaudeBot (Anthropic) | Model training | 96 | 18.6% | 126 |
| Meta-ExternalAgent (Meta) | Model training | 94 | 18.2% | 93 |
| Google-Extended (Google) | Model training | 83 | 16.1% | 110 |
| Applebot-Extended (Apple) | Model training | 78 | 15.1% | 82 |
| PerplexityBot (Perplexity) | AI search index | 72 | 14.0% | 118 |
| Claude-SearchBot (Anthropic) | AI search index | 57 | 11.0% | 76 |
| OAI-SearchBot (OpenAI) | AI search index | 49 | 9.5% | 94 |
| Perplexity-User (Perplexity) | Fetch when a user asks | 64 | 12.4% | 69 |
| Claude-User (Anthropic) | Fetch when a user asks | 60 | 11.6% | 76 |
| ChatGPT-User (OpenAI) | Fetch when a user asks | 58 | 11.2% | 104 |
| Googlebot (Google) | Control: Google Search | 12 | 2.3% | 65 |
The seven training crawlers are each blocked by 15% to 22% of sites. The three search crawlers are blocked by 9.5% to 14%. OpenAI shows the gap most clearly: 97 sites block GPTBot and 49 block OAI-SearchBot. PerplexityBot is the most blocked search crawler at 14.0%, and it is named in more files (118) than any agent except GPTBot and ClaudeBot.
“Named in the file” counts sites that write a rule group for that agent by name, whether to allow it or block it. A site can block an agent without naming it, through a User-agent: * group that disallows everything. That is why Bytespider and Meta-ExternalAgent show more blocks than mentions. Googlebot is the control: 12 sites (2.3%) disallow it, almost all of them sites that block every crawler they do not name, such as reddit.com.
The common policy: keep training out, let search in
This section answers how sites combine the rules. Of the 136 sites that block at least one training crawler, 62 leave every AI search crawler we tested allowed. Spotify, Medium and Canva all do this: GPTBot and ClaudeBot are disallowed, OAI-SearchBot, Claude-SearchBot and PerplexityBot are not. Only one site in the sample blocks an AI search crawler while allowing every training crawler.
At the other end, 47 sites block all three AI search crawlers. They include amazon.com, facebook.com, nytimes.com, bbc.com, cnn.com, reddit.com, pinterest.com and tiktok.com. Those sites have opted out of being fetched for AI search answers altogether. TikTok blocks Bytespider, the crawler run by its own parent company.
In all, 175 of the 516 sites (33.9%) name at least one AI agent in robots.txt. Two thirds of the top sites have written no AI-specific rule at all, so every AI agent falls under their general rules.
Which AI agents can read your site?
The free audit runs these checks on your domain, plus the CDN and firewall blocks that robots.txt does not show.
Get a Free AI Visibility AuditPrefer to talk it through? Book a Free Call
Blocking by rank: the top 100 block most
This section answers whether the biggest sites behave differently. They block more, and they publish llms.txt less.
| Rank band | Block GPTBot | Block OAI-SearchBot | Publish llms.txt |
|---|---|---|---|
| Ranks 1–100 | 24.1% (13 of 54) | 16.7% (9 of 54) | 14.0% (8 of 57) |
| Ranks 101–500 | 17.7% (34 of 192) | 7.8% (15 of 192) | 25.0% (55 of 220) |
| Ranks 501–1,000 | 18.5% (50 of 270) | 9.3% (25 of 270) | 23.1% (71 of 308) |
Among the top 100, 24.1% block GPTBot and 16.7% block OAI-SearchBot, against 17.7% and 7.8% for ranks 101 to 500. The top band is small (54 sites with a robots.txt), so treat the gap as a direction, not a precise figure.
llms.txt: almost one in four top sites publish one
This section answers how widely the llms.txt proposal has spread. llms.txt is a plain-text or markdown file at the site root that summarises a site for language models (what llms.txt is). No AI vendor requires it. Even so, 134 of the 585 top websites (22.9%) serve one, and 127 of those open with a markdown title as the proposal describes. Adoption is highest among developer and software companies: cloudflare.com, github.com, stripe.com, shopify.com, salesforce.com, hubspot.com and paypal.com all publish one.
In the top 100 only 8 of 57 websites (14.0%) publish one, against 25.0% for ranks 101 to 500. The very largest consumer platforms are the least likely to have one.
Inside the files: nine more findings
This section answers what the robots.txt and llms.txt files say beyond a simple yes or no. Findings 1 to 8 come from a second reading of the same 516 robots.txt files. Finding 9 comes from fetching the llms.txt files again on November 14, 2026.
1. Blocking training is not blocking ChatGPT
42 of the 97 sites that disallow GPTBot still allow ChatGPT-User, the agent ChatGPT sends when a person asks it to read a page. The same split shows at Google: 71 of the 83 sites that block Google-Extended still allow Googlebot, as Google intends. Sites treat OpenAI and Anthropic the same way. Only 12 block GPTBot and not ClaudeBot, and 11 do the reverse.
2. Rules for names Anthropic no longer lists
71 sites write a rule for anthropic-ai and 64 for Claude-Web. Neither name appears in Anthropic’s current crawler list, which names ClaudeBot, Claude-User and Claude-SearchBot. Most of these sites also name ClaudeBot. Five name only the old tokens, including cloudflare.com, chatgpt.com and espn.com. On those sites ClaudeBot falls under the general * rules, which may not be what the authors meant. Copied blocklists go out of date when vendors rename their crawlers.
3. Some sites open only the homepage
Our headline figures count a site as blocking an agent only when its root / is disallowed. Eleven more sites let GPTBot fetch the homepage and nothing else. They allow /$, which matches the root only, and disallow everything else. chatgpt.com, vimeo.com, soundcloud.com, naver.com and statista.com follow this pattern. Tested at an inner page, GPTBot is blocked on 108 sites (20.9%) and OAI-SearchBot on 58 (11.2%). That makes the headline figures a lower bound.
4. 27 sites only let in the crawlers they name
27 sites, including facebook.com, linkedin.com, reddit.com, pinterest.com, imdb.com and reuters.com, disallow every crawler they do not name and then list the ones they accept. AI agents are blocked on these sites unless they are on the list. Five sites put AI agents on it. netflix.com accepts all three OpenAI and all three Anthropic agents, plus PerplexityBot and Google-Extended. wsj.com accepts OpenAI’s three agents while ClaudeBot and PerplexityBot fall under its block-everything rule.
5. A new line: Content-Signal
22 sites (4.3%) add a Content-Signal line, a format introduced by Cloudflare that says what fetched content may be used for: search, AI input or AI training. All 22 say yes to search. Twelve say no to AI training, among them patreon.com, twilio.com, wiley.com and unesco.org. cloudflare.com itself says yes to all three. The line states a preference; it does not block anything.
6. Long blocklists and empty files
We checked 29 AI agent names in all. 329 sites (63.8%) name none of them, and 40 name 20 or more. The median robots.txt is 1.2 KB with two rule groups. The largest is figma.com’s at 386 KB. who.int has the most rule groups, 527. 367 sites (71.1%) list a sitemap, and 61 set a crawl delay.
7. Allowing an agent is not the same as allowing everything
Some sites name an AI agent and allow it. 55 do this for GPTBot and 62 for OAI-SearchBot. Most of them still keep that agent out of some paths: 28 of the 55 GPTBot groups and 41 of the 62 OAI-SearchBot groups disallow at least one path while allowing the site root.
8. Sites with an llms.txt block AI crawlers less
16 of the 133 sites with both files (12.0%) block at least one AI training crawler. Among the 383 sites with a robots.txt and no llms.txt, 120 do (31.3%). Sites that write a file for language models are mostly sites that want language models to read them. GitHub is an exception: it publishes a 28 KB llms.txt and blocks Bytespider.
9. What is inside the llms.txt files
131 of the 134 llms.txt files were still online when we fetched them again on November 14. 109 of the 131 follow the proposal’s shape: a markdown title, then a one-paragraph summary in a blockquote. The median file is 13 KB with 65 links under 9 section headings. 37 include the proposal’s “Optional” section for links an assistant may skip. 32 link to markdown copies of their pages (.md URLs). Seven contain no links at all. The companion file llms-full.txt, which carries full page text, is much rarer: 21 of 585 websites (3.6%) serve one, including cloudflare.com, github.com, shopify.com and digitalocean.com.
16 more AI agents in the same files
This section answers how the top sites treat AI agents beyond the 13 above. We applied the same RFC 9309 test to 16 more names found in AI blocklists. They are evaluated against the same 516 files, so the shares compare directly with the main table.
| Agent | Blocked | Share | Named in the file |
|---|---|---|---|
| Diffbot (Diffbot) | 89 | 17.2% | 72 |
| omgili (Webz.io) | 88 | 17.1% | 70 |
| cohere-ai (Cohere) | 83 | 16.1% | 77 |
| ImagesiftBot (ImageSift) | 74 | 14.3% | 52 |
| anthropic-ai (Anthropic; not in Anthropic’s current list) | 72 | 14.0% | 71 |
| Claude-Web (Anthropic; not in Anthropic’s current list) | 71 | 13.8% | 64 |
| FacebookBot (Meta) | 71 | 13.8% | 59 |
| Timpibot (Timpi) | 71 | 13.8% | 50 |
| Amazonbot (Amazon) | 70 | 13.6% | 70 |
| YouBot (You.com) | 69 | 13.4% | 60 |
| AI2Bot (Allen Institute for AI) | 64 | 12.4% | 44 |
| DuckAssistBot (DuckDuckGo) | 58 | 11.2% | 56 |
| Meta-ExternalFetcher (Meta) | 57 | 11.0% | 44 |
| MistralAI-User (Mistral) | 56 | 10.9% | 40 |
| GoogleOther (Google) | 51 | 9.9% | 40 |
| Applebot (Apple; Siri and Spotlight search) | 21 | 4.1% | 37 |
Apple shows the purpose split most clearly. Applebot, which powers Siri and Spotlight search, is blocked by 21 sites (4.1%). Applebot-Extended, which only controls whether Apple may train on the content, is blocked by 78 (15.1%). Amazon’s Amazonbot is blocked by 70 sites (13.6%), more than any of the AI search crawlers in the main table except PerplexityBot.
What this means for a business website
This section answers what to do with these numbers if you run a smaller site.
- Decide per job, not per company. Blocking GPTBot does not remove a site from ChatGPT search answers; OpenAI documents that OAI-SearchBot controls that, independently. The same split exists at Anthropic and Perplexity. If you want to be cited by assistants, the search agents must be allowed.
- Google-Extended is not a crawler. It is a product token for Gemini training and grounding, and Google states it does not affect Google Search. We answered the common question about AI Overviews in Does Google-Extended affect AI Overviews?
- Check the CDN, not just the file. robots.txt only states a request. In our agency study, three of 80 sites with permissive robots files still refused AI agents at the firewall. The ten-minute check is in Is your CDN blocking AI crawlers?
- Write the rules on purpose. Two thirds of top sites have no AI-specific rule at all. Our AI crawler checker shows what your robots.txt allows for each agent, and which AI crawlers visit your site explains what each one does.
Method
This section answers exactly how the numbers were produced, so they can be checked or repeated.
- Sample. The top 1,000 domains of the Tranco list Y897G, the research ranking that combines several traffic sources and resists manipulation.
- Websites only. We requested each homepage with a desktop Chrome user agent. 279 domains did not respond (DNS, CDN and API domains such as akamaiedge.net), 73 returned an error and served no robots.txt even after a retry with full browser headers and the www host, and 5 served something other than a web page. 41 sites refused the homepage request but served robots.txt; they stay in, because leaving them out would remove exactly the sites with bot protection. Domains that land on the same site (fbcdn.net and facebook.com) were counted once: 58 duplicates removed, leaving 585 websites.
- robots.txt. 516 websites served a robots.txt file, which is the base for every blocking figure. 48 returned 4xx (no rules, so everything is allowed), 18 returned an HTML page and 3 an error.
- Verdicts. Each file was evaluated under RFC 9309: the agent’s own group if one names it, otherwise the
*group; the longest matching rule wins and Allow wins a tie. “Blocked” means the site root/is disallowed. - llms.txt. Present means
/llms.txton the final host returned 200 with a body of at least 50 bytes that is neither HTML nor JSON.
All requests were made on November 13, 2026 (05:05 UTC on November 14), one request per file.
Limitations
- Only root blocks count. A site that disallows its articles but not its homepage is counted as allowing that agent; finding 3 above measures this (11 more sites for GPTBot).
- The llms.txt contents (finding 9) come from a second fetch on November 14, 2026; presence figures come from the first.
- robots.txt is a request. Whether each crawler honours it, and whether a CDN blocks it anyway, is not measured here.
- One fetch from one US location. Some sites serve different files by region or user agent; we did not test variants.
- Sites that refused even the robots.txt request (adobe.com, w3.org and cdc.gov among them) are excluded, which probably understates blocking slightly.
- The top-100 band holds 54 sites with a robots.txt; differences between bands are directional.
Download the data and cite this study
This section gives you the per-site results and a ready citation. The CSV has one row per website: Tranco rank, domain, final host, robots.txt and llms.txt status, and an allowed or blocked verdict for each of the 14 agents. It is free to reuse under CC BY 4.0.
APA
Webvello Research. (2026, November 13). Who blocks AI crawlers? We read the robots.txt of the top 1,000 websites. https://www.webvello.com/research/who-blocks-ai-crawlers
One line, for articles
Source: Webvello Research, robots.txt study of the top 1,000 websites, November 2026 (https://www.webvello.com/research/who-blocks-ai-crawlers)
Questions about this study
What share of top websites block AI crawlers?
Of 516 top-1,000 websites that served a robots.txt on November 13, 2026, 136 (26.4%) disallowed at least one AI training crawler and 75 (14.5%) disallowed at least one AI search crawler. The rest named no AI agent or allowed them.
Which AI crawler is blocked most often?
ByteDance’s Bytespider (113 sites, 21.9%) and Common Crawl’s CCBot (109, 21.1%), followed by OpenAI’s GPTBot (97, 18.8%) and Anthropic’s ClaudeBot (96, 18.6%). OpenAI’s search crawler OAI-SearchBot was the least blocked AI agent (49, 9.5%).
Does blocking GPTBot keep a site out of ChatGPT search?
No. OpenAI documents GPTBot and OAI-SearchBot as separate agents with independent settings: GPTBot governs training, OAI-SearchBot governs inclusion in ChatGPT search answers. 62 top sites in this study block training crawlers while allowing every AI search crawler.
How many top websites have an llms.txt file?
134 of 585 websites in the top 1,000 (22.9%) served an llms.txt file on November 13, 2026, counting only plain-text or markdown files of at least 50 bytes. 127 of them open with a markdown title, as the proposal describes.
If a site blocks GPTBot, can ChatGPT still open its pages?
Often, yes. Of the 97 top sites that disallow GPTBot, 42 still allow ChatGPT-User, the agent ChatGPT uses to fetch a page when a person asks about it (November 2026). Each OpenAI agent follows its own robots.txt group.
Do sites with an llms.txt block AI crawlers less?
Yes. 16 of 133 top websites with both files (12.0%) disallowed an AI training crawler, against 120 of 383 sites without an llms.txt (31.3%), in November 2026.
Can I use this data?
Yes. The per-site CSV is free to reuse under CC BY 4.0. Credit “Webvello Research” and link to this page.
More from Webvello Research
Find out which AI agents can read your site
A free audit covers robots.txt, CDN blocks, structured data and how your pages read to an assistant, with a written plan of what to fix first.