AI crawlers and robots.txt

Search crawlers and training crawlers

The vendors publish the distinction because site owners were treating “AI” as one switch. A search crawler fills an answer product’s index. A training crawler collects text that may be used to train a model. A user-triggered fetcher opens a page because a person asked the assistant to look at that URL. The three are not synonyms. The rules below are the ones each company printed, as read on 12 October 2026. Tokens and version numbers move. When you change a rule, re-read the page you are about to cite.

How the rules interact

OpenAI says each robots.txt setting is independent. You can allow OAI-SearchBot and disallow GPTBot. If you allow both, OpenAI may use one crawl for both jobs so it does not fetch twice. A change can take about 24 hours to be picked up. The full strings and the IP ranges are on OpenAI’s crawler documentation. OAI-SearchBot’s example string includes OAI-SearchBot/1.4. GPTBot’s includes GPTBot/1.4. ChatGPT-User’s is ChatGPT-User/1.0. OpenAI says the version number may change.

ChatGPT-User is the one people misread. OpenAI says it is not an automatic crawl, that robots.txt may not apply because a person initiated the fetch, and that it is not used to decide whether content appears in Search. The search opt-out is OAI-SearchBot. Blocking ChatGPT-User in robots.txt and believing you have left search is the wrong conclusion. Blocking OAI-SearchBot and believing user-triggered fetches will also stop is the other wrong conclusion.

Anthropic’s note, dated 7 April 2026, says ClaudeBot, Claude-User, and Claude-SearchBot honor robots.txt, including a Crawl-delay if you send one. The opt-out has to be in robots.txt on each host. Anthropic says blocking the IP addresses instead may not stick, because the crawler then cannot read the file that asks it to stay away. IP ranges are published at claude.com/crawling/bots.json, linked from Anthropic’s crawler article.

Perplexity’s crawler docs say PerplexityBot and Perplexity-User are independent, and that a robots.txt change can take up to 24 hours. Perplexity-User generally ignores robots.txt. The help-center page, modified 14 September 2026, adds two limits. Perplexity does not build foundation models, so allowing PerplexityBot is not a training grant. A disallowed site’s full text is not indexed, but the domain, a headline, and a short factual summary may still be. Perplexity also says an older feature, in which a user could ask it to summarize a URL that robots.txt disallowed, has been turned off.

A firewall can undo the file

robots.txt is a request, read by the crawler that is able to fetch it. OpenAI and Perplexity both say to allow their published IP ranges as well as the token. A bot-protection rule that challenges or drops those addresses will keep the crawler out even when the file says Allow. The page can look open in a browser and closed in the search index.

Verify a visitor before you write a rule from a log line. Anyone can put Googlebot or PerplexityBot in a user-agent header. Google, OpenAI, Anthropic, and Perplexity each publish addresses or a verification method on the pages linked above. A spoofed header is not a reason to disallow the real token.

A conservative public-site policy

For a site that wants to be found, the usual policy is: allow Googlebot, allow the search tokens you want to be eligible for, and decide the training tokens on purpose. A blanket Disallow under User-agent: * does not mean what people hope. A crawler that matches a more specific group uses that group. A crawler you never named falls through to the star. Name the tokens you have an opinion about.

Do not disallow Googlebot to “turn off AI”. AI Overviews are part of Google Search. The snippet controls and the Search Console inclusion setting are the documented limits, described on Google AI Overviews and AI Mode. Do not disallow OAI-SearchBot and then expect a page to be cited from Chat GPT’s search index.

Training opt-outs are a policy choice, not an SEO tactic. Disallowing GPTBot tells OpenAI not to use the content for training foundation models. It is not a ranking boost, and OpenAI does not say it removes knowledge already in a trained model. Make the choice in writing, in the file, and keep it consistent across subdomains.

What robots.txt cannot do

It cannot force a citation. An allowed crawler may still never fetch the URL, or fetch it and not name it. It cannot hide a page from a person who has the link. It cannot fix a page that never answers the question. The writing is How to write a page AI search can cite. The difference between this file and an llms.txt map is What is llms.txt?.