Quick answers

AIGrow provides AI visibility monitoring, a business assistant, blog publishing and scoped automation services. Start with the free scan to inspect the output before paying.

Run the free scan

The scan is free and monitoring plans start at $29 a month. The operations audit is $290, the visibility audit is $490, and custom builds receive a written price before work starts.

See the plans

Run the free scan. It reads your site, asks several AI assistants what your customers ask, and points you at one thing. No signup required, and it will tell you if you do not need us.

Start now

Yes. Use the contact form for product, billing, support or project questions. Describe the goal and the system involved so the first reply can be specific.

Contact AIGrow
AI robots.txt generator

AI robots.txt generator

Choose which AI crawlers may read your site. Start with a preset, change any crawler, and download the AI part of your robots.txt. Free, with no account needed.

Free and without sign-up. Your choices stay in this page’s URL so you can share them.

Preset

OAI-SearchBot

OpenAI: ChatGPT search

Claude-SearchBot

Anthropic: Claude search

PerplexityBot

Perplexity: Perplexity

Applebot

Apple: Siri and Spotlight

DuckAssistBot

DuckDuckGo: DuckDuckGo answers

Amazonbot

Amazon: Alexa and Rufus

YouBot

You.com: You.com answers

User crawlers

ChatGPT-User

OpenAI: ChatGPT

Claude-User

Anthropic: Claude

Perplexity-User

Perplexity: Perplexity

MistralAI-User

Mistral: Le Chat

Meta-ExternalFetcher

Meta: Meta AI

Training crawlers

GPTBot

OpenAI: future OpenAI models

ClaudeBot

Anthropic: future Claude models

Google-Extended

Google: Gemini training and grounding

Applebot-Extended

Apple: Apple Intelligence training

Meta-ExternalAgent

Meta: Meta AI

CCBot

Common Crawl: the open dataset most models train on

Bytespider

ByteDance: Doubao

cohere-ai

Cohere: Cohere models

Googlebot and Bingbot are not in the file. They also crawl for Google Search and Bing, so your User-agent: * group keeps deciding for them. To keep your pages out of Gemini training and grounding, block Google-Extended.

Optional. One path per line, up to 20 lines. Start each path with / and use no spaces or #. These paths are included in the shared URL.

Your AI robots.txt rules

# aigrow.gg/tools/ai-robots-txt

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: Meta-ExternalFetcher
User-agent: Amazonbot
User-agent: YouBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
Disallow: /

Put these rules in robots.txt at the root of your site and keep any existing User-agent: * group.

Check what your site lets in now

This free AI robots.txt generator writes rules for the crawlers in the AIGrow directory. By default, it lets search and user request crawlers in and keeps training crawlers out. Choose a different preset or change individual crawlers, add paths to exclude, then copy or download the rules for your site.

How this works

Choose a starting point
The default allows search crawlers and crawlers that open a page on request, and blocks training crawlers. You can allow every crawler, block every crawler, or change each one separately.
One group for the same rules
Crawlers with the same rules share a group with several User-agent lines. The names come from the AI crawler directory. Googlebot and Bingbot are left out because they also crawl for Google Search and Bing.
Paths apply to every group
A crawler named in a group does not inherit the User-agent: * group, as RFC 9309 describes. Extra Disallow paths are added to every AI group. Other paths stay allowed for the crawlers you let in.
Keep the rest of your file
The generator writes only the AI groups. Keep your existing User-agent: * group and other entries, including sitemap links. Replace any old groups naming these same crawlers, and carry over the paths you want to keep excluded.

Questions about AI crawlers

It writes User-agent groups with Allow or Disallow rules for the crawlers in the AI crawler directory, apart from Googlebot and Bingbot, which also crawl for Google Search and Bing. Crawlers with the same rules share a group. You can copy the text or download it as robots.txt, then add it to your existing file.

They do different jobs and have different names. Search crawlers build the index an assistant searches. User request crawlers open a page when someone asks. Training crawlers collect pages for future models, so you can choose rules for them separately.

robots.txt is a request to crawlers, described in the operators’ documentation. The file states the rules you ask them to use.

It goes at the root of your site, for example https://example.com/robots.txt. If you already have a file there, keep its User-agent: * group and other entries, and update the AI groups with the generated rules.

Can AI assistants read your business pages?

Run the free scan to check access and see what assistants say about your business.

Run the free AI scan