How Does ChatGPT Search Pick Sources? What OpenAI Documents

Most writing about “ranking in ChatGPT” is confident about things OpenAI has never said. The documentation that does exist is narrow and useful: it tells you which agent decides whether your site can appear in ChatGPT search, which agent is about training, which one shows up when a person asks about you, and the consequence of blocking each. Start there, because an error at this layer makes everything downstream irrelevant.

What OpenAI actually documents

OpenAI’s crawler documentation describes its agents by job and is explicit about consequences. Three of them matter to a business website.

OAI-SearchBot: the agent that decides search inclusion

OpenAI says OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features.” The consequence is stated just as plainly: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers,” although they can still appear “as navigational links.” In other words, a blocked site might be linked when someone asks for it by name, but it will not be drawn on to answer a question. Robots.txt changes take “~24 hours” to propagate, and the IP ranges are published at openai.com/searchbot.json.

ChatGPT-User: a person, not a crawler

“When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent.” OpenAI stresses that this agent “is not used for crawling the web in an automatic fashion,” and that “because these actions are initiated by a user, robots.txt rules may not apply.” It also “is not used to determine whether content may appear in Search.” Treat it as a browser: if your firewall returns a 403 or a challenge page to it, the person on the other end does not get your page in their answer.

GPTBot: training, and only training

GPTBot “is used to crawl content that may be used in training our generative AI foundation models.” OpenAI describes each setting as “independent of the others,” which is the sentence that resolves most robots.txt confusion: blocking GPTBot is a training decision and, on the published documentation, has no effect on whether ChatGPT search can show you. The reverse is also true. Allowing GPTBot does nothing for your visibility if OAI-SearchBot is blocked.

What OpenAI does not document

The documentation covers access. It does not describe how ChatGPT search ranks the pages it can reach, what signals it weights, how fresh its index is, or what that index is built from. So any article that tells you ChatGPT prefers a particular word count, schema type, domain age or backlink profile is describing an inference, a test or a guess. Some of those guesses may be right. None of them are documented, and we are not going to present them as if they were.

What we can say with confidence is structural. ChatGPT search retrieves pages and composes an answer from them. A page it cannot retrieve cannot contribute. A page it retrieves but cannot parse into a clear statement about who you are and what you do is a weak contributor. Those two observations are enough to order the work.

Four things that follow

1. Allow OAI-SearchBot explicitly and keep the training decision separate. The robots.txt below does that in three visible groups so the next person to edit it understands what each line costs. Our robots.txt generator writes the same groups.
2. Confirm a normal 200, not just an allow rule. A firewall or bot-management default can challenge an agent your robots.txt permits. Request your homepage and your top service pages with the OAI-SearchBot and ChatGPT-User user-agent strings and compare the status codes to a browser request. Then grep your logs for both tokens over the last month and verify source addresses against the published JSON lists. The wider version of this check, across every vendor, is in which AI crawlers visit your site.
3. Be indexed and unambiguous. A retrieval system can only use pages it has, and can only attribute them to a business it can identify. One organization node, one set of facts, the same name, address, phone and services everywhere. If your page says who you are only in a hero image or after client-side scripts run, assume it is not saying it at all.
4. Give every key page a quotable answer and test with a prompt panel. Write the answer to the page’s question in the first 60 words, back it with a table or a list, then ask ChatGPT your ten most common customer questions in fresh sessions each month and record whether you are named. The diagnostic version of that process, ordered from cheapest check to hardest, is in why isn’t ChatGPT mentioning my business.

A robots.txt that keeps the three decisions apart

# Required if you want to appear in ChatGPT search answers
User-agent: OAI-SearchBot
Allow: /

# Fetches a page when a person asks about it; robots.txt may not apply
User-agent: ChatGPT-User
Allow: /

# Training only. Independent of search. Allow or Disallow on your terms.
User-agent: GPTBot
Allow: /

User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml

Change the GPTBot block to Disallow: / if your policy is no training. Leave OAI-SearchBot and ChatGPT-User alone unless you have decided you do not want to appear in ChatGPT answers, and remember the roughly 24-hour delay before testing any change.

Where this fits

Access is the first of four things every assistant needs from a site; identity, quotable answers and off-site corroboration are the other three, and the order we work them in is set out in the AI Search Playbook. For the commercial side, our guides to getting recommended by ChatGPT and ChatGPT for local businesses pick up where the documentation stops, and say so when they do.

Frequently asked questions

About the work

Webvello is an AI search optimization and index engineering company founded in 2024, based in Roseville, California. We’ve delivered 50+ projects across 37+ cities, with average client ROI in the 3–5x range on properties where the technical foundation was the binding constraint. Founder George Shvaya came to search from cybersecurity and systems engineering, which is why this article reads the vendor’s documentation and server logs instead of repeating what everyone assumes about ChatGPT. That discipline is the starting point of our generative engine optimization work.

Source: OpenAI’s crawler documentation, linked above and re-read on October 16, 2026. OpenAI adds agents and changes behavior; check the linked page before relying on a statement here.