Is Your CDN or Firewall Blocking AI Crawlers? A 10-Minute Check

Most of the sites we audit that are invisible to AI assistants have done nothing wrong on purpose. Somebody turned on a bot-protection feature in 2023, or accepted a default in 2025, and the setting has been quietly returning 403s to the agents that would have quoted the site ever since. Nothing in Search Console reports it. Nothing in analytics reports it. The only evidence is in the CDN’s firewall log, which nobody reads.

Why this became a default problem

Cloudflare fronts a large share of the web, and on July 1, 2025 it announced it was “changing the default to block AI crawlers unless they pay creators for their content.” Alongside it came pay per crawl, described as a private beta, in which a crawler without payment intent receives a 402 Payment Required response with pricing and publishers can Allow, Charge or Block each crawler.

For a publisher that sells content, that is a reasonable position. For a plumber, a dental practice or a software vendor whose entire goal is to be recommended, blocking the agents that do the recommending is self-defeating. The problem is that the default does not distinguish between a crawler collecting training data and an agent fetching your page because a customer just asked about you. Generic WAF bot rules, “bot fight” modes and managed challenges have the same blind spot: anything that is not a mainstream browser gets the same treatment.

Two decisions, not one

The vendors themselves document their agents separately, which makes the policy question tractable. Decide first whether you want to be cited. If yes, the search and user-initiated agents need to get a 200. Decide second, as a matter of policy, whether your content may be used for training. That is what the training crawlers and the Google-Extended token control, and nothing about that decision needs to touch the first one.

Sources: OpenAI, Anthropic, Perplexity and Google. OpenAI is explicit that sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers,” and that ChatGPT-User “is not used to determine whether content may appear in Search.” Google is equally explicit that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.” For the full roster and what each agent reads, see which AI crawlers visit your site.

The four-step check

Step 1: Read your robots.txt

Fetch /robots.txt and look for rules naming any of the agents above, and for a catch-all User-agent: * block. Check that the file itself returns a 200 to a bot user-agent and not a challenge page; a crawler that cannot read your rules cannot honour them. If you need to rewrite it, the robots.txt generator produces per-agent blocks so the search agents and training crawlers can be treated differently.

Step 2: Fetch a page as each agent

Robots.txt is advisory. The firewall is not. The only way to know what an agent actually receives is to ask as that agent and compare the response to a normal browser request:

# Baseline: what a browser gets
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/130 Safari/537.36" https://example.com/services/ | head -1

# Repeat for each agent string
for UA in "OAI-SearchBot/1.0; +https://openai.com/searchbot" \
          "ChatGPT-User/1.0; +https://openai.com/bot" \
          "Claude-SearchBot" "Claude-User" \
          "PerplexityBot/1.0; +https://perplexity.ai/perplexitybot" \
          "Perplexity-User/1.0; +https://perplexity.ai/perplexity-user"; do
  printf "%-60s " "$UA"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $UA)" https://example.com/services/
done

Every line should match the baseline. A 403, a 429, a 503, or a 200 whose body is a “checking your browser” page instead of your content, is a block. Run it against the homepage, a service page and /robots.txt. If you cannot see your firewall configuration, this test is still available to you, which is the point of it.

Step 3: Search the logs for the last 30 days

Grep your origin access logs for the agent strings. If they appear with 200s, the agents are reaching you. If they do not appear at all, that is not proof they never came: a CDN that blocks at the edge never forwards the request, so the origin log is silent. Open the CDN’s own firewall or security events view and filter by the same strings; the blocks, challenges and 402s are recorded there with the rule that fired.

grep -hE "OAI-SearchBot|ChatGPT-User|GPTBot|Claude-SearchBot|Claude-User|ClaudeBot|PerplexityBot|Perplexity-User" access.log* \
  | awk '{print $9}' | sort | uniq -c   # status code histogram per agent family

Step 4: Verify by IP where the vendor publishes one

User-agent strings can be spoofed, which is why some firewalls are suspicious of them. Each vendor publishes the addresses its agents use: OpenAI at openai.com/searchbot.json and openai.com/chatgpt-user.json, Anthropic at claude.com/crawling/bots.json, Perplexity at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json. Perplexity’s own advice for WAF rules is to “combine both User-Agent string matching and IP address verification” and refresh the lists periodically. Build the allow rule from name plus published range, and you have stopped both the false block and the spoofed request.

One warning from Anthropic belongs here. Its documentation discourages IP-level blocking because it may prevent its bots from reading robots.txt at all, so an IP block does not reliably opt you out; it just makes your intentions unreadable. The same logic applies in reverse to allow rules: if the agent cannot reach /robots.txt, nothing you wrote there is in force.

Why the user-initiated agents matter most

Two of the agents in the table behave differently from the rest. OpenAI says that when users ask ChatGPT a question “it may visit a web page with a ChatGPT-User agent,” and that because the action was initiated by a user, “robots.txt rules may not apply.” Perplexity says Perplexity-User “generally ignores robots.txt rules” for the same reason. These are not crawlers filling an index on a schedule. They are a person, right now, asking about the thing you sell, and the assistant reaching for your page to answer them.

A 403 or a challenge page returned to that fetch means the assistant cannot quote you for that user, at that moment. It will answer anyway, from whatever it could read, which is usually a competitor or a directory page about you that you do not control. Of all the responses in your firewall log, those are the ones that cost a sale each.

Symptoms, causes and fixes

What this does not fix

Access is a precondition, not a result. An agent that can now reach your page still has to find it in an index, read it as HTML, understand who you are, and find a sentence worth quoting. Those are the next checks, and they are laid out in order in why ChatGPT is not mentioning your business. Google’s guidance for its own AI features is a useful reminder of how little magic is involved: a page must be indexed and eligible for a snippet, and there are “no additional technical requirements” and no special structured data to add. Access, then index coverage, then clarity. In that order.

Frequently asked questions

About the work

Webvello is an AI search optimization and index engineering company founded in 2024, based in Roseville, California. We have delivered 50+ projects across 37+ cities, with a 3–5x average client ROI on properties where the technical foundation was the binding constraint. Founder George Shvaya came to search from cybersecurity and systems engineering, which is why a firewall log is the first thing we open on a new engagement. The rest of the method is in the AI Search Playbook.