Robots.txt tester

Test up to 50 paths with the robots.txt tester. Compare search and AI crawler rules, check blocked assets and sitemaps, then edit and retest a draft.

Free, no account. Ten site checks per hour.

robots.txt to test

Our server reads /robots.txt on this host. The file can be cached for five minutes.

A path or a full address. Empty means the address above.

Up to 50 paths. When filled, these replace the single path above. Full URLs must belong to the live file’s site.

A robots.txt tester tells you whether a crawler may fetch a URL under a site’s robots.txt rules. This one reads the live file or a pasted draft and matches it the way RFC 9309 and Google describe: the group for that crawler applies, the longest matching rule wins and allow wins a tie. It shows the deciding line.

How to use this checker

Read the live file or paste a draft
Enter a site to read its live /robots.txt, or paste a draft to test it before you publish. A pasted draft goes to our server only to run the match, and nothing is fetched for it.
Choose the path and the crawler
Enter one path or up to 50 paths, one per line. Compare Googlebot, Bingbot and all listed AI tokens in one matrix. Named groups are merged; otherwise wildcard rules apply. Full URLs must belong to the live file’s site.
Read the deciding rule
The longest matching pattern wins and Allow wins a tie. Wildcards and end anchors follow Google’s matcher behavior. The first 500 KiB is read. A 4xx except 429 allows crawling; server errors use disallow all without assuming a previous cached file.

Questions about this checker

A 404 or another 4xx answer means there are no rules, so every crawler may fetch everything. A 5xx answer, a timeout or a bot check page is different: RFC 9309 tells crawlers to treat the whole site as disallowed until the file can be read.

No. Disallow stops crawling, and a blocked page can still be indexed from links to it, without a description. To keep a page out of search, allow crawling and add a noindex meta tag or an X-Robots-Tag header.

The major operators say they do, and GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended each have a published token. Robots.txt only binds crawlers that choose to follow it, so use firewall rules too if you need a hard block.

Google does not support Crawl-delay. Bing and some other crawlers read it. The tester flags the line so you know it changes nothing for Googlebot.

Related tools

  • AI robots.txt generator

    Use this AI robots.txt generator to choose which AI crawlers may read your site. Edit a preset and download the AI rules to merge with your existing file.

  • Robots.txt generator

    Use this robots.txt generator to write the file for your whole site. Start from a WordPress, Shopify, Wix or Squarespace default, then test any path against the result.

  • AI Overviews checker

    Use this AI Overviews checker to review a page's Googlebot access, indexing and snippet controls. Read the findings before changing access settings.

Check your business in AI answers

Technical checks help you repair a page. The free AI visibility scan shows the answers checked for your business and what to work on next.

Run the free AI visibility scan