AI robots.txt generator

Use this AI robots.txt generator to choose which AI crawlers may read your site. Edit a preset and download the AI rules to merge with your existing file.

Free and without sign-up. Your choices stay in this page’s URL so you can share them.

Preset

OAI-SearchBot

OpenAI: ChatGPT search

Claude-SearchBot

Anthropic: Claude search

PerplexityBot

Perplexity: Perplexity

Applebot

Apple: Siri and Spotlight

DuckAssistBot

DuckDuckGo: DuckDuckGo answers

MistralAI-Index

Mistral: Vibe search

Meta-WebIndexer

Meta: Meta AI search

Amzn-SearchBot

Amazon: Alexa search

YouBot

You.com: You.com answers

User crawlers

ChatGPT-User

OpenAI: ChatGPT

Claude-User

Anthropic: Claude

Perplexity-User

Perplexity: Perplexity

MistralAI-User

Mistral: Vibe

Amzn-User

Amazon: Alexa

Training crawlers

GPTBot

OpenAI: future OpenAI models

ClaudeBot

Anthropic: future Claude models

Google-Extended

Google: Gemini training and grounding

Applebot-Extended

Apple: Apple Intelligence training

MistralAI-Training

Mistral: future Mistral models

Meta-ExternalAgent

Meta: Meta AI

Amazonbot

Amazon: Amazon products and models

CCBot

Common Crawl: the open dataset most models train on

Bytespider

ByteDance: Doubao

cohere-ai

Cohere: legacy robots.txt preference

Googlebot and Bingbot are not in the file. They also crawl for Google Search and Bing, so your User-agent: * group keeps deciding for them. To keep your pages out of Gemini training and grounding, block Google-Extended.

Optional. One path per line, up to 20 lines. Start each path with / and use no spaces or #. These paths are included in the shared URL.

Your AI robots.txt rules

# aigrow.gg/tools/ai-robots-txt

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalFetcher
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: YouBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: MistralAI-Training
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
Disallow: /

Put these rules in robots.txt at the root of your site and keep any existing User-agent: * group.

Check what your site lets in now

We can also set robots.txt, your CDN rules and your sitemap for you, as you choose, and check them again after. See the AI crawler access setup

This free AI robots.txt generator writes rules for the crawlers in the AIGROW directory. By default, it lets search and user request crawlers in and keeps training crawlers out. Choose a different preset or change individual crawlers, add paths to exclude, then copy or download the rules for your site.

How this works

Choose a starting point
The default allows search crawlers and crawlers that open a page on request, and blocks training crawlers. You can allow every crawler, block every crawler, or change each one separately.
One group for the same rules
Crawlers with the same rules share a group with several User-agent lines. The names come from the AI crawler directory. Googlebot and Bingbot are left out because they also crawl for Google Search and Bing.
Paths apply to every group
A crawler named in a group does not inherit the User-agent: * group, as RFC 9309 describes. Extra Disallow paths are added to every AI group. Other paths stay allowed for the crawlers you let in.
Keep the rest of your file
The generator writes only the AI groups. Keep your existing User-agent: * group and other entries, including sitemap links. Replace any old groups naming these same crawlers, and carry over the paths you want to keep excluded.

Questions about AI crawlers

It writes User-agent groups with Allow or Disallow rules for the crawlers in the AI crawler directory, apart from Googlebot and Bingbot, which also crawl for Google Search and Bing. Crawlers with the same rules share a group. You can copy the text or download it as robots.txt, then add it to your existing file.

They do different jobs and have different names. Search crawlers build the index an assistant searches. User request crawlers open a page when someone asks. Training crawlers collect pages for future models, so you can choose rules for them separately.

robots.txt is a request to crawlers, described in the operators’ documentation. The file states the rules you ask them to use.

It goes at the root of your site, for example https://example.com/robots.txt. If you already have a file there, keep its User-agent: * group and other entries, and update the AI groups with the generated rules.

Can AI assistants read your business pages?

Run the free scan to check access and see what assistants say about your business.