We Fetched 81 Marketing Agency Homepages as the AI Search Agents

Why we ran it

In July we validated the structured data of 81 marketing agencies. This study takes the same 81 domains and asks the question that comes before structured data: can the AI search agents fetch the page at all, and when they do, does the page say who the business is? We ran our AI visibility audit engine against each homepage on October 16, 2026. The method and its limits are at the end.

Access: robots.txt is not where the blocking happens

Each homepage was requested as a normal browser and then as each of the six agents OpenAI, Anthropic and Perplexity document for search indexing and user-initiated fetches. robots.txt was parsed per agent with the longest-match rule.

The robots column is the striking one. Agencies that sell visibility have not opted out of it: not one disallows a search agent, a user agent, or even a training crawler. 79 of 80 serve a robots.txt at all; one has none, which means everything is allowed by default.

The failures are elsewhere. Three sites answered a browser with 200 and answered at least one AI agent with a 403 or a refused connection. One refused all six agents. Two refused only OpenAI’s two agents, one with a 403 and one by dropping the connection, while serving the Anthropic and Perplexity agents normally. That is the signature of a CDN or firewall rule keyed on the user-agent string, and it is invisible in robots.txt. OpenAI documents that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers; a 403 has the same effect without the opt-out ever being chosen. The ten-minute check for this is in Is your CDN blocking AI crawlers?

Every one of the 80 delivered its main content in server HTML; none fell below our 150-word threshold for visible text without JavaScript. Agencies, at least, are not shipping client-only homepages.

Identity: the page that cannot say who it is

Reachable is necessary, not sufficient. The second group of checks asks whether the page states, in structured data and in the HTML, who the business is.

Forty percent have an Organization or LocalBusiness node with a name and some way to contact the business. Another 42% have the node but nothing an assistant could use to corroborate it: no address, no phone, no email. Eighteen percent have no organization node at all on the homepage. Two sites ship JSON that does not parse, which disables the whole block.

Meanwhile 46% publish an llms.txt file. No vendor requires it, and it is cheap, so adoption is not surprising. But a plain-text description of the company is a weak substitute for a machine-readable identity that matches the visible page. The order should be the other way round. What llms.txt is and is not is in the glossary.

Quotability: half the sample opens with something an assistant can use

Forty of 80 homepages open with a first substantive paragraph between 25 and 120 words, which is the shape of a sentence an assistant can lift. Thirty-six open with something too short to say anything (a slogan) or too long to extract from (a wall). Four have no paragraph of fifteen words anywhere in the delivered HTML. Headings are not the problem: 78 of 80 have three or more section headings. The gap is between having sections and having an answer.

Scores

The audit weights access most, then identity, then quotability, then measurement. On that scale the median agency homepage scored 94 of 100, the lowest 80, and 64 of 80 scored 90 or above. That is a reasonable result for a group that sells this work, and it says something useful: the basics are mostly in place, and the remaining failures are specific and identifiable per site rather than general.

What to do with this if you are not an agency

1. Check the CDN, not just robots.txt. Request your homepage as each agent and compare the status code to a browser. The AI crawler checker does it in one click.
2. Give the organization node a body. Name, URL, address, telephone, email, one @id, referenced from every page. This is the single most common gap in the sample.
3. Open with the sentence. Forty to sixty words that say who you are, what you do, where and for whom. Half of professional marketing sites do not have it.
4. Add llms.txt last, not first. It costs nothing, but it does not replace the two items above.

The full order of operations is in the AI Search Playbook, and the twelve checks a complete audit should run are in What should a GEO audit include?

Methodology and limits

• Sample. The same 81 agency domains used in the July 2026 structured data study. 80 returned HTTP 200 to a browser request; one returned a response larger than the 3 MB fetch limit and was excluded.
• Fetches. One request per agent per homepage on October 16, 2026, using the user-agent strings the vendors document, with a ten-second timeout and manual redirect following. “No response” means a connection error or timeout; we did not retry. A first pass used a smaller size limit for the agent probes than for the browser fetch, which produced three false failures on large homepages; the numbers here are from a second pass with identical limits.
• Robots. robots.txt parsed per agent group with longest-match semantics; the verdict is whether the site root is disallowed for that agent. Path-level rules were not scored.
• Identity and quotability. Homepage HTML parsed server-side; JSON-LD parsed as JSON; the checks are the ones listed in the tables. Homepages only; service pages were not audited.
• What this does not measure. Whether any assistant cites these sites, IP-based CDN rules that pass user-agent matching, or anything about ranking. Google AI Overviews and AI Mode follow Googlebot and ordinary indexing, which were evaluated only through robots.txt and the noindex check.
• Reproducibility. The engine is the one behind the free audit; anyone can run the same checks on any domain. Raw per-domain results are kept in our research archive and are not published to avoid naming individual businesses.

Frequently asked questions

About the work

Webvello is an AI search optimization and index engineering company founded in 2024, based in Roseville, California, with 50+ projects across 37+ cities. Founder George Shvaya came to search from cybersecurity and systems engineering, which is why this study measures what can be observed rather than what can be claimed.