AI crawlers
AI crawlers are automated clients that fetch web content for AI related products. Their purposes differ: GPTBot and ClaudeBot collect material for model development, while OAI-SearchBot and Claude-SearchBot serve search. User-triggered clients fetch pages on request. A robots.txt rule for one token does not automatically control every client from the same operator.
Why it matters for a business
An owner may want search discovery while declining training collection. Those choices need separate rules. The server, firewall and robots.txt can also give different signals. A user-agent test from AIGROW cannot prove how the operator’s real IP addresses are treated.
Example and evidence
The AIGROW crawler directory groups clients by search, training and user-request purpose. Its check reports robots.txt preferences separately from representative HTTP requests. An allowed preference with a blocked request is a reason to inspect the server, not proof that an assistant can read the page.
Compare crawler purposes and tokens
What to do next
Review the operator’s current documentation before changing rules. State each relevant product token explicitly where needed. Check the page path, response status and any indexing directives. Treat public access tests as diagnostics and keep private content behind authentication.
Common question
Related terms
Sources and method
Sources checked on 2026-10-09. Examples marked illustrative are not measured business results.
Check your own business in AI answers
Run the free scan to see which assistants name your business and which facts need attention. A scan is a sample, not a promise of future recommendations.