AI Crawler Checker
The AI crawler checker reads your robots.txt and then fetches your homepage as each of the six search and user agents that OpenAI, Anthropic and Perplexity document. It tells you, per agent, whether you are blocked by a rule, blocked by your CDN, or reachable. About twenty seconds, nothing stored.
The agents, and what each one decides
Summarized from each vendor’s documentation, linked per row.
| Agent | Operator | Documented role | Source |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Decides whether a site can appear in ChatGPT search answers. | docs |
| ChatGPT-User | OpenAI | Fetches a page when a person asks ChatGPT about it; robots.txt may not apply. | docs |
| GPTBot | OpenAI | Training only; documented as independent of search. | docs |
| Claude-SearchBot | Anthropic | Indexes pages to improve the relevance and accuracy of search results. | docs |
| Claude-User | Anthropic | Fetches a page when a person asks Claude; disabling it may reduce visibility. | docs |
| ClaudeBot | Anthropic | Collects content for training. | docs |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search; not used for training. | docs |
| Perplexity-User | Perplexity | Fetches a page when a user asks; generally ignores robots.txt. | docs |
| Googlebot | Governs Search including AI Overviews and AI Mode; no separate agent to allow. | docs | |
| Google-Extended | A product token for Gemini training and grounding; no separate user agent; does not affect Search inclusion. | docs |
How to read the result
- Blocked by robots.txt. The rule is yours to change. Separate the search and user agents from the training crawlers; the vendors document them as independent decisions.
- No response or a non-200 status while a browser gets 200. Your CDN or firewall is answering for you. Allow the search and user agents and verify by the vendor IP lists rather than user-agent string alone.
- Reachable everywhere but still not cited. Access is only the first of four needs. The full audit checks identity and quotability, and the AI Search Playbook sets out the order to fix things in.
Blocked somewhere and not sure where?
The free audit runs the access, identity and quotability checks and lists a fix for every finding.
Common questions
Which AI crawlers does this tool check?
It fetches your homepage as the six search and user-initiated agents the vendors document: OAI-SearchBot and ChatGPT-User from OpenAI, Claude-SearchBot and Claude-User from Anthropic, PerplexityBot and Perplexity-User from Perplexity. It also evaluates robots.txt for Googlebot and for the training crawlers GPTBot, ClaudeBot and the Google-Extended token.
What does "blocked by robots.txt" mean?
The robots.txt group that applies to that agent disallows the site root. OpenAI documents that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, and Anthropic documents that disabling Claude-User may reduce visibility in user-directed web search. Training crawlers are a separate decision and the vendors describe them as independent of search visibility.
My robots.txt allows the agent but the live fetch failed. Why?
A CDN, firewall or bot-management rule is answering before your server does. A 403, a 429 or a challenge page to a user-initiated agent means the assistant cannot read the page for that user. Check the bot rules in your CDN and verify against the vendor IP lists.
Does this check Google AI Overviews?
Google documents that AI Overviews and AI Mode follow ordinary Googlebot crawling and indexing, with no separate agent to allow. The tool evaluates robots.txt for Googlebot and checks that the homepage is not marked noindex; the full audit covers the rest.
Is anything stored?
No. The check runs on request and the result is shown to you only. Nothing is saved unless you ask for the written audit on the full audit page.