Robots.txt generator

Use this robots.txt generator to write the file for your whole site. Start from a WordPress, Shopify, Wix or Squarespace default, then test any path against the result.

Free, no account. The file is built in your browser and nothing is sent.

Start from

Every crawler may request every path. Add the paths to keep out below.

Optional. Used for the sitemap lines and the robots.txt tester link.

Group 1

* stands for every crawler without a group of its own. Other examples: Googlebot, Bingbot.

Paths to keep out, one per line, such as /admin/ or /*?sort=.

Exceptions inside a disallowed path, one per line.

Optional. Google ignores it. Bing reads 1 to 30 seconds.

Full addresses. Sitemap lines apply to every crawler.

Your robots.txt

Read back by the parser: 1 group, 0 rules, 0 sitemap lines, 64 bytes.

# aigrow.gg/tools/robots-txt-generator

User-agent: *
Disallow:

Test a path

Allowed Googlebot has no group of its own and follows the * group. No rule matches this path, so it is allowed.

A robots.txt file sits at the root of a site and tells crawlers which paths they may request. This generator writes user agent groups, allow and disallow rules and sitemap lines, then reads the file back with the RFC 9309 parser our robots.txt tester uses, so you can test a path before you upload it.

How this works

Groups and rules
Each group names one or more user agents, the paths they may not request (Disallow) and the exceptions (Allow). A path starts with / or *, and $ marks the end of a URL. A crawler follows the group that names it and falls back to the * group when none does.
Platform defaults
The WordPress preset matches the file WordPress writes itself. The Shopify, Wix and Squarespace presets copy the default files that sites on those platforms served in October 2026. Shopify edits go through the robots.txt.liquid theme template and Wix has a robots.txt editor in its SEO settings. Squarespace documents no way to upload your own file, so its preset is for reference.
Crawl delay
Google ignores Crawl-delay. Bing reads it as a time window of 1 to 30 seconds. The generator takes whole seconds from 1 to 30 and writes the line only in the groups where you set it.
Checked before you copy
The finished file is read back with the RFC 9309 parser our robots.txt tester runs. If a line would be read differently from how it was meant, or the file passes Google’s 500 KiB limit, the page says so. The path test shows which line decides for a given crawler.

Questions about robots.txt

No. It stops crawling, and Google can still index a blocked URL without its content when other pages link to it. To keep a page out of results, let it be crawled and add a noindex robots meta tag or an X-Robots-Tag header.

At the root of the host, for example https://example.com/robots.txt. Each subdomain and each protocol needs its own file, and a file in a folder is ignored. On Shopify, Wix and Squarespace the platform serves the file, so changes go through their settings or theme.

A group needs at least one rule line. An empty Disallow allows everything, and it stops the next group’s user agents from joining this one.

Google and RFC 9309 pick the most specific rule, the one with the longest path. When an Allow and a Disallow rule are equally long, Allow wins. The path test shows the deciding line.

Add a group for each crawler, or use the AI robots.txt generator, which lists the AI crawlers by purpose and writes their lines. Add those lines below your existing groups.

Related tools

  • Robots.txt tester

    Test up to 50 paths with the robots.txt tester. Compare search and AI crawler rules, check blocked assets and sitemaps, then edit and retest a draft.

  • AI robots.txt generator

    Use this AI robots.txt generator to choose which AI crawlers may read your site. Edit a preset and download the AI rules to merge with your existing file.

  • SEO checker

    Use this SEO checker to audit up to 25 pages for crawl and on-page issues. Review affected URLs and a fix list, or add a small competitor sample.

Check your business in AI answers

Pages that crawlers can read and follow are the groundwork. The free AI visibility scan shows how assistants answer about your business and what to work on next.

Run the free AI visibility scan