Robots.txt for AI search: which bots to allow
To appear in AI search answers, allow the search crawlers in your robots.txt file. Search crawlers and training crawlers are separate. You can allow ChatGPT, Claude or Perplexity to read your site for search, and still block the bot each company uses for model training. A blanket ban on all bots will keep you out of those answers. Allowing a crawler does not guarantee a mention. AI results vary.
The table and examples below follow each company's published docs. Check those pages if a bot name changes.
Why does robots.txt matter for AI search?
When someone asks ChatGPT, Claude or Perplexity who to use, the assistant can search the web and cite pages. If its search crawler is blocked, your pages are much less likely to be used in those answers.
OpenAI's crawler documentation is explicit: sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. It recommends allowing OAI-SearchBot, and says a robots.txt change can take about 24 hours to take effect for search. Perplexity says the same 24-hour lag for its crawlers. Some website builders, security plugins and firewalls also block bots at the server, even when robots.txt allows them, so check those settings too.
This is one of the first things to fix if you are asking why your business is not showing up on ChatGPT.
Which bot is for search, and which is for training?
Each company publishes more than one user agent. The search bot is the one that affects whether you can be named in answers. The training bot is a separate choice.
| Company | Search or indexing bot | Training bot | User-triggered fetcher |
|---|---|---|---|
| OpenAI | OAI-SearchBot, used to surface sites in ChatGPT search | GPTBot, used for training generative AI foundation models | ChatGPT-User, used when a person asks ChatGPT to visit a page. OpenAI says this is not an automatic crawl, robots.txt rules may not apply, and it is not used to decide Search inclusion. |
| Anthropic | Claude-SearchBot, used to index content for search | ClaudeBot, used to collect content that may be used for training | Claude-User, used when a person asks Claude to retrieve a page. Anthropic says disabling it may reduce visibility for user-directed web search, and that its bots honour robots.txt. |
| Perplexity | PerplexityBot, used to surface and link sites in Perplexity search. Perplexity says it is not used to crawl content for foundation models. | None published under a separate training name | Perplexity-User, used when a person asks Perplexity to visit a page. Perplexity says this fetcher generally ignores robots.txt, because a user requested the fetch. |
OpenAI also has OAI-AdsBot, which only visits pages submitted as ads and, according to OpenAI, is not used to train foundation models. It is not the bot that decides whether you appear in ChatGPT search.
For Google Search, including AI Overviews and AI Mode, the crawler is Googlebot. Google's common crawlers documentation says preferences addressed to Googlebot affect Google Search and its features. Google-Extended is a separate robots.txt token that controls whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says Google-Extended does not affect inclusion in Google Search and is not used as a ranking signal. Blocking Google-Extended does not replace allowing Googlebot if you want to show up in Google's AI features. See Google AI Overviews for local businesses.
What should a typical robots.txt file include?
If you want to be eligible for AI search answers, and you prefer not to allow training crawls, a clear starting point is:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Allow: /If you are happy for your public pages to be used in training as well, change those Disallow rules to Allow: /, or leave the training bots out so the rest of your file governs them.
Do not add a User-agent: * / Disallow: / rule unless you intend to block every crawler that does not have its own group. If that blanket rule is already in your file, or was added by a security plugin, it can keep OAI-SearchBot, Claude-SearchBot and PerplexityBot out until you give those bots their own allow rules.
Anthropic says it honours a Crawl-delay directive for its bots. OpenAI and Perplexity do not document the same directive in the pages cited above, so do not assume it applies to them.
Put the file at https://yourdomain.com/robots.txt, on every hostname you care about (including www and any booking or blog subdomain).
What if a firewall is still blocking the crawler?
robots.txt only works if the crawler can fetch it and then fetch the pages. OpenAI, Anthropic and Perplexity each publish IP ranges for their bots and tell site owners not to rely on blocking those addresses if they want the crawler to read the file. Perplexity also publishes notes for web application firewalls, because a firewall can challenge or block the bot even when robots.txt allows it.
If ChatGPT, Claude or Perplexity still never cites your site after a robots.txt change, check your host, CDN or security plugin for a bot-blocking rule, and look at your server logs for the official user-agent strings.
How do you check that the change worked?
Open /robots.txt in a browser and confirm the rules are the ones you intended. Wait at least a day, then ask a customer-style question in a fresh ChatGPT, Claude or Perplexity chat with search on, and see whether your pages appear in the citations. Being cited is not the same as being named, and AI results vary, so use this only as a crawl check. The method for counting a recommendation is in how to get your business recommended by ChatGPT and on the method page.
Google's guide to generative AI features is the place to start for Google's own AI search features, which depend on ordinary Search crawling rather than on OAI-SearchBot. That distinction is also covered in GEO vs SEO for local businesses.
Frequently asked questions
Should I block all AI bots to protect my content?
That is your choice for training crawlers. If you also block the search crawlers, ChatGPT, Claude and Perplexity are much less likely to use your pages in answers. The two decisions are separate.
Does allowing OAI-SearchBot mean ChatGPT will name my business?
No. It only means your site can be used in ChatGPT search answers. AI results vary, and many other sources can be cited instead of you.
Do user-triggered fetchers follow robots.txt?
Not always. OpenAI says ChatGPT-User may not follow robots.txt because the visit is initiated by a user, and that this bot is not used to decide Search inclusion. Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says Claude-User does honour robots.txt, and that disabling it may reduce visibility for user-directed search.
Will a robots.txt change show up immediately?
OpenAI and Perplexity both say to allow about 24 hours. Other delays can come from caches, CDNs or a firewall that is still blocking the bot.
Need this as part of a wider check?
Allowing the search crawlers is one step in making your business easier to name. The method page explains how I test customer questions, and generative engine optimisation sets out how that work sits alongside ordinary search. AI results vary.