Quick answers

AIGrow provides AI visibility monitoring, a business assistant, blog publishing and scoped automation services. Start with the free scan to inspect the output before paying.

Run the free scan

The scan is free and monitoring plans start at $29 a month. The operations audit is $290, the visibility audit is $490, and custom builds receive a written price before work starts.

See the plans

Run the free scan. It reads your site, asks several AI assistants what your customers ask, and points you at one thing. No signup required, and it will tell you if you do not need us.

Start now

Yes. Use the contact form for product, billing, support or project questions. Describe the goal and the system involved so the first reply can be specific.

Contact AIGrow
AI crawlers

AI crawlers: what each bot does and what blocking it changes

The crawlers that read websites for AI assistants: who runs each one, what its work feeds, and the robots.txt lines to allow or block it.

Run the free crawler check

AI crawlers are the bots that read websites for AI assistants. Search crawlers such as OAI-SearchBot build the index an assistant cites, user crawlers such as ChatGPT-User open a page when someone asks, and training crawlers such as GPTBot collect pages for future models. This list covers 22 of them, with robots.txt rules for each.

Three kinds of AI crawler

Search
Search crawlers read pages to build the index an assistant searches when it answers. A page they cannot read cannot be found or cited in those answers.
User request
User crawlers open one page when a person asks an assistant to read it. A page they cannot read cannot be opened in that conversation.
Training
Training crawlers collect pages for future models. Keeping them out does not stop the crawlers that fetch pages for answers, which have names of their own.

Every crawler, by what it feeds

Each name is the robots.txt token. Open one for its user agent and rules.

Search crawlers

CrawlerOperatorWhat it feeds
OAI-SearchBotOpenAIChatGPT search
Claude-SearchBotAnthropicClaude search
PerplexityBotPerplexityPerplexity
GooglebotGoogleGoogle Search and AI Overviews
BingbotMicrosoftBing and Copilot
ApplebotAppleSiri and Spotlight
DuckAssistBotDuckDuckGoDuckDuckGo answers
AmazonbotAmazonAlexa and Rufus
YouBotYou.comYou.com answers

User crawlers

CrawlerOperatorWhat it feeds
ChatGPT-UserOpenAIChatGPT
Claude-UserAnthropicClaude
Perplexity-UserPerplexityPerplexity
MistralAI-UserMistralLe Chat
Meta-ExternalFetcherMetaMeta AI

Training crawlers

CrawlerOperatorWhat it feeds
GPTBotOpenAIfuture OpenAI models
ClaudeBotAnthropicfuture Claude models
Google-ExtendedGoogleGemini training and grounding
Applebot-ExtendedAppleApple Intelligence training
Meta-ExternalAgentMetaMeta AI
CCBotCommon Crawlthe open dataset most models train on
BytespiderByteDanceDoubao
cohere-aiCohereCohere models

A robots.txt template

robots.txt is a plain text file at the root of a site (yoursite.com/robots.txt). A group starts with one or more User-agent lines that name crawlers, followed by Allow and Disallow lines for paths. A crawler named in a group follows the groups that name it and ignores the group for every crawler (User-agent: *).

These lines let in every crawler that fetches pages for answers. The training crawlers follow behind # signs: letting them in is your choice.

Add these lines to the robots.txt file at the root of the site, and remove any group that disallows these crawlers. A crawler named in its own group follows only that group, so copy into it any Disallow lines it should keep.

# Crawlers that fetch pages for answers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: Meta-ExternalFetcher
User-agent: Amazonbot
User-agent: YouBot
Allow: /

# Training crawlers: your choice. Remove the # signs to let them in.
# User-agent: GPTBot
# User-agent: ClaudeBot
# User-agent: Google-Extended
# User-agent: Applebot-Extended
# User-agent: Meta-ExternalAgent
# User-agent: CCBot
# User-agent: Bytespider
# User-agent: cohere-ai
# Allow: /

Build a full AI robots.txt

Which of them does your site let in?

robots.txt is one way to turn a crawler away. A CDN or firewall setting is another, and the site still looks fine in a browser. The free crawler check reads your robots.txt and requests your page as each crawler, in a few seconds.

Run the free crawler check

Questions about AI crawlers

Both are run by OpenAI. OAI-SearchBot is a search crawler: it reads pages for ChatGPT search. GPTBot is a training crawler: it collects pages for future OpenAI models. robots.txt can block one and allow the other.

Add a group that names the crawler, or lists several, followed by “Disallow: /”. A crawler named in a group follows the groups that name it, so the rules for every crawler (User-agent: *) no longer apply to it.

Training crawlers and the crawlers that fetch pages for answers have different names. A site can keep the training crawlers out and still let the answering ones in.

The free crawler check reads your robots.txt for all 22 crawlers in this list and requests a page as each one that has a user agent, then shows which are let in and which are turned away.

Can AI assistants read your site?

The free crawler check answers in a few seconds, with no account and no email.

Run the free crawler checkRun the free scan