AI crawler checker

Use this AI crawler checker to review robots.txt rules, indexing signals and HTTP responses. Inspect blocked access before changing your site's rules.

Free, with no account and no email. The check takes several seconds.

A homepage or any other page. Only that page is checked.

This free AI crawler access checker reads robots.txt, checks meta robots and X-Robots-Tag signals, and sends requests using AI crawler user agents. Each row shows the matching rule and HTTP response. Allowed, blocked and unclear results describe what we observed from our server; they cannot prove that a real crawler will get the same response.

How this check works

robots.txt
The site’s robots.txt is read once and each crawler’s rules for the page are applied as RFC 9309 describes: the group that names the crawler wins over the group for every crawler.
A request as each crawler
The page is requested using each crawler’s user agent. The table shows the HTTP status, any challenge and the agent string sent. Google-Extended and Applebot-Extended are control tokens with no separate request. Googlebot and Bingbot are checked in robots.txt only.
Control request
The page is also requested as an ordinary browser. When that request is refused too, the crawler requests say nothing about crawlers in particular, and only robots.txt is judged.
The limit
Our requests come from our address. A firewall that checks a crawler’s published IP addresses can refuse our copy and still admit the real crawler, so Googlebot and Bingbot are judged by robots.txt only.

Every crawler in the table has its own page in the AI crawler list.

Build a full AI robots.txt · Check title and description length

Questions about AI crawlers

Enter your domain or a full page address and select Check the page. Read the status for each crawler, inspect the matching robots.txt line and HTTP response, then fix unwanted restrictions and check again. Only the submitted page is tested. Results can be cached for ten minutes.

Noindex asks search engines to leave a page out of their index. Noai and noimageai express usage preferences with varying operator support. None of these tags prevents an HTTP request. This checker shows the observed signals separately from robots.txt and firewall responses.

Search crawlers, such as OAI-SearchBot, index content for answers. User crawlers fetch pages when someone asks. Training crawlers collect content for model development. You can choose training preferences separately. Google-Extended also controls Gemini grounding, and some user-triggered fetchers may ignore robots.txt.

A CDN, a firewall or a bot protection service can refuse requests by user agent before robots.txt is ever read. Some have a setting for AI crawlers. The site still opens in a browser, so the block goes unnoticed.

No. llms.txt is a proposed file that lists a site’s main pages for language models. The crawlers in this check need robots.txt to allow them and the server to serve the page; an llms.txt file changes neither.

No. Allowing them only means the assistants can read the page. Whether a business is named depends on what the page and other sites say about it.

See what the assistants say about you

The free scan reads your site, checks the same crawlers and asks assistants the questions your buyers ask.