AI crawlers: what each bot does and what blocking it changes

The crawlers that read websites for AI assistants: who runs each one, what its work feeds, and the robots.txt lines to allow or block it.

AI crawlers are the bots that read websites for AI assistants. Search crawlers such as OAI-SearchBot build the index an assistant cites, user crawlers such as ChatGPT-User open a page when someone asks, and training crawlers such as GPTBot collect pages for future models. This list covers 27 of them, with robots.txt rules for each.

Three kinds of AI crawler

Search
Search crawlers read pages to build the index an assistant searches when it answers. A page they cannot read cannot be found or cited in those answers.
User request
User crawlers open one page when a person asks an assistant to read it. A page they cannot read cannot be opened in that conversation.
Training
Training crawlers collect pages for future models. Keeping them out does not stop the crawlers that fetch pages for answers, which have names of their own.

Every crawler, by what it feeds

Each name is the robots.txt token. Open one for its user agent and rules.

Search crawlers

CrawlerOperatorWhat it feeds
OAI-SearchBotOpenAIChatGPT search
Claude-SearchBotAnthropicClaude search
PerplexityBotPerplexityPerplexity
GooglebotGoogleGoogle Search and AI Overviews
BingbotMicrosoftBing and Copilot
ApplebotAppleSiri and Spotlight
DuckAssistBotDuckDuckGoDuckDuckGo answers
MistralAI-IndexMistralVibe search
Meta-WebIndexerMetaMeta AI search
Amzn-SearchBotAmazonAlexa search
YouBotYou.comYou.com answers

User crawlers

CrawlerOperatorWhat it feeds
ChatGPT-UserOpenAIChatGPT
Claude-UserAnthropicClaude
Perplexity-UserPerplexityPerplexity
MistralAI-UserMistralVibe
Meta-ExternalFetcherMetaMeta AI
Amzn-UserAmazonAlexa

Training crawlers

CrawlerOperatorWhat it feeds
GPTBotOpenAIfuture OpenAI models
ClaudeBotAnthropicfuture Claude models
Google-ExtendedGoogleGemini training and grounding
Applebot-ExtendedAppleApple Intelligence training
MistralAI-TrainingMistralfuture Mistral models
Meta-ExternalAgentMetaMeta AI
AmazonbotAmazonAmazon products and models
CCBotCommon Crawlthe open dataset most models train on
BytespiderByteDanceDoubao
cohere-aiCoherelegacy robots.txt preference

A robots.txt template

robots.txt is a plain text file at the root of a site (yoursite.com/robots.txt). A group starts with one or more User-agent lines that name crawlers, followed by Allow and Disallow lines for paths. A crawler named in a group follows the groups that name it and ignores the group for every crawler (User-agent: *).

These lines let in every crawler that fetches pages for answers. The training crawlers follow behind # signs: letting them in is your choice.

Add these lines to the robots.txt file at the root of the site, and remove any group that disallows these crawlers. A crawler named in its own group follows only that group, so copy into it any Disallow lines it should keep.

# Crawlers that fetch pages for answers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalFetcher
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: YouBot
Allow: /

# Training crawlers: your choice. Remove the # signs to let them in.
# User-agent: GPTBot
# User-agent: ClaudeBot
# User-agent: Google-Extended
# User-agent: Applebot-Extended
# User-agent: MistralAI-Training
# User-agent: Meta-ExternalAgent
# User-agent: Amazonbot
# User-agent: CCBot
# User-agent: Bytespider
# User-agent: cohere-ai
# Allow: /

Build a full AI robots.txt

Which of them does your site let in?

robots.txt is one way to turn a crawler away. A CDN or firewall setting is another, and the site still looks fine in a browser. The free crawler check reads your robots.txt and requests your page as each crawler, in a few seconds.

Run the free crawler check

We can also check your robots.txt, CDN rules and sitemap for you and set them as you choose. See the AI crawler access setup

Questions about AI crawlers

Both are run by OpenAI. OAI-SearchBot is a search crawler: it reads pages for ChatGPT search. GPTBot is a training crawler: it collects pages for future OpenAI models. robots.txt can block one and allow the other.

Add a group that names the crawler, or lists several, followed by “Disallow: /”. A crawler named in a group follows the groups that name it, so the rules for every crawler (User-agent: *) no longer apply to it.

Training crawlers and the crawlers that fetch pages for answers have different names. A site can keep the training crawlers out and still let the answering ones in.

The free crawler check reads your robots.txt for all 27 crawlers in this list and requests a page as each one that has a user agent, then shows which are let in and which are turned away.

Can AI assistants read your site?

The free crawler check answers in a few seconds, with no account and no email.