Which AI crawlers should I allow?

If you want to appear in AI search, allow the crawlers that serve search surfaces: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot and Claude-SearchBot from Anthropic, PerplexityBot from Perplexity, and Google-Extended for Google's generative surfaces. Whether you also allow training-only crawlers such as CCBot is a separate decision and does not directly affect your search visibility.

What happens if I block an AI crawler in robots.txt?

Your chance of appearing as a source in that engine's answers largely disappears. If the model cannot fetch your page, the quality of your content changes nothing. This is the most common technical reason behind complaints about missing from AI search, and it cannot be fixed by editing content.

Do user-triggered fetchers obey robots.txt?

No. Fetchers such as ChatGPT-User, Perplexity-User and Google-Agent retrieve a single page because a person asked for it. By design they do not consult robots.txt, because the request is made on a user's behalf rather than by a crawler. You cannot block them with robots.txt; server-side access controls are needed if you must.

How do I verify my robots.txt actually works?

Confirm that yoursite.com/robots.txt opens as plain text, then request a page with a crawler user-agent and check that it returns 200. A 200 alone is not enough if your page is drawn only with JavaScript: the content must be present in the server-rendered HTML, because AI crawlers do not execute JavaScript.

// GEONI Guides

AI Crawler List and robots.txt Rules (2026)

Q: Should I add llms.txt too?

It is optional. Google states plainly that llms.txt is not consulted by its search or generative surfaces. It costs little, so you may add it for non-Google systems, but it should not be presented as a ranking or citation lever for Google.

Last updated: 6 August 2026 · GEONI Team

The most common technical reason for missing from AI search is access, not content. If the model cannot fetch your page, the quality of the text on it changes nothing. Below: the main AI crawlers, what they come for, and a copy-paste allow block.

The main AI crawlers

User-agent	Owner	What it fetches for	Obeys robots.txt
`GPTBot`	OpenAI	Crawling / model data	Yes
`OAI-SearchBot`	OpenAI	Search surface	Yes
`ChatGPT-User`	OpenAI	Single page, user-triggered	No (user-triggered)
`ClaudeBot`	Anthropic	Crawling	Yes
`Claude-SearchBot`	Anthropic	Search surface	Yes
`PerplexityBot`	Perplexity	Search index	Yes
`Perplexity-User`	Perplexity	Single page, user-triggered	No (user-triggered)
`Google-Extended`	Google	Opt-in switch for generative surfaces	Yes
`CCBot`	Common Crawl	Open dataset	Yes
`anthropic-ai`	Anthropic	Legacy / general identifier	Yes

Note: Googlebot is a separate crawler serving classic search; Google-Extended does not affect it.

Copy-paste allow block

Add this to yoursite.com/robots.txt. Always keep a sitemap line at the end — crawlers take your page list from there.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

User-triggered fetchers are a separate class

ChatGPT-User, Perplexity-User and similar agents fetch a page because a person asked for that address. It is a single request made on a user's behalf, not a crawl, and by design these do not consult robots.txt. You cannot block them there; server-side access control is the only route if you genuinely need to.

Allowing access is not enough on its own

A 200 is good but insufficient. AI crawlers do not execute JavaScript. If your page is drawn only in the browser by JS, the crawler sees an empty shell. The content must exist in the server-rendered HTML.

Check three things together: (1) does robots.txt allow it, (2) does a request with a crawler user-agent return 200, (3) does the returned HTML contain real text.

Verify it: two commands

# 1) Does robots.txt open, and is it plain text?
curl -s https://yoursite.com/robots.txt

# 2) What does the page return to a crawler user-agent?
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; GPTBot/1.0)" https://yoursite.com/

Anything other than 200 (403, 503, a redirect) means bot protection or a WAF is in the way, and your provider needs an exception for AI crawlers.

Is llms.txt required?

Optional. Google states plainly that llms.txt is not consulted by its search or generative surfaces. It costs little, so you may add it for non-Google systems — but it should not be sold as a ranking or citation lever for Google. Detail: what is llms.txt.

Frequently asked questions

Do I have to allow all of them?

No. Allowing the search-serving crawlers is enough for AI visibility; blocking training-only crawlers is a separate choice and does not directly affect it.

Does blocking protect my content?

robots.txt is a courtesy protocol, not an access control. It stops the crawlers that respect it — not the ones that don't, and not user-triggered requests.

How soon does a change take effect?

Crawlers cache robots.txt; changes usually land within days. Pages appearing in answers additionally requires crawl and index time.

Let's measure whether crawlers can read your site

A GEONI scan checks your robots.txt rules, crawler access and whether real text exists in your server-rendered HTML — then shows where you stand across four AI engines. Your first scan is free.

Run a free scan →