AI crawler checker
Use this AI crawler checker to review robots.txt rules, indexing signals and HTTP responses. Inspect blocked access before changing your site's rules.
Free, with no account and no email. The check takes several seconds.
This free AI crawler access checker reads robots.txt, checks meta robots and X-Robots-Tag signals, and sends requests using AI crawler user agents. Each row shows the matching rule and HTTP response. Allowed, blocked and unclear results describe what we observed from our server; they cannot prove that a real crawler will get the same response.
How this check works
- robots.txt
- The site’s robots.txt is read once and each crawler’s rules for the page are applied as RFC 9309 describes: the group that names the crawler wins over the group for every crawler.
- A request as each crawler
- The page is requested using each crawler’s user agent. The table shows the HTTP status, any challenge and the agent string sent. Google-Extended and Applebot-Extended are control tokens with no separate request. Googlebot and Bingbot are checked in robots.txt only.
- Control request
- The page is also requested as an ordinary browser. When that request is refused too, the crawler requests say nothing about crawlers in particular, and only robots.txt is judged.
- The limit
- Our requests come from our address. A firewall that checks a crawler’s published IP addresses can refuse our copy and still admit the real crawler, so Googlebot and Bingbot are judged by robots.txt only.
Every crawler in the table has its own page in the AI crawler list.
Build a full AI robots.txt · Check title and description length
Questions about AI crawlers
See what the assistants say about you
The free scan reads your site, checks the same crawlers and asks assistants the questions your buyers ask.