AI robots.txt generator
Use this AI robots.txt generator to choose which AI crawlers may read your site. Edit a preset and download the AI rules to merge with your existing file.
Free and without sign-up. Your choices stay in this page’s URL so you can share them.

Your AI robots.txt rules
# aigrow.gg/tools/ai-robots-txt
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalFetcher
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: YouBot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: MistralAI-Training
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
Disallow: /
Put these rules in robots.txt at the root of your site and keep any existing User-agent: * group.
Check what your site lets in now
We can also set robots.txt, your CDN rules and your sitemap for you, as you choose, and check them again after. See the AI crawler access setup
This free AI robots.txt generator writes rules for the crawlers in the AIGROW directory. By default, it lets search and user request crawlers in and keeps training crawlers out. Choose a different preset or change individual crawlers, add paths to exclude, then copy or download the rules for your site.
How this works
- Choose a starting point
- The default allows search crawlers and crawlers that open a page on request, and blocks training crawlers. You can allow every crawler, block every crawler, or change each one separately.
- One group for the same rules
- Crawlers with the same rules share a group with several User-agent lines. The names come from the AI crawler directory. Googlebot and Bingbot are left out because they also crawl for Google Search and Bing.
- Paths apply to every group
- A crawler named in a group does not inherit the User-agent: * group, as RFC 9309 describes. Extra Disallow paths are added to every AI group. Other paths stay allowed for the crawlers you let in.
- Keep the rest of your file
- The generator writes only the AI groups. Keep your existing User-agent: * group and other entries, including sitemap links. Replace any old groups naming these same crawlers, and carry over the paths you want to keep excluded.
Questions about AI crawlers
Can AI assistants read your business pages?
Run the free scan to check access and see what assistants say about your business.