AI crawlers
cohere-ai: a training crawler run by Cohere
cohere-ai is a training crawler. It collects pages for this: Cohere models. Operator: Cohere.
At a glance
- Operator
- Cohere
- robots.txt token
cohere-ai- Purpose
- Training. Training crawlers collect pages for future models. Keeping them out does not stop the crawlers that fetch pages for answers, which have names of their own.
- What it feeds
- Cohere models
What blocking it changes
If you keep cohere-ai out, your pages are left out of what it collects for this: Cohere models. The crawlers that fetch pages for answers have names of their own, so they can still read your site.
robots.txt lines for cohere-ai
robots.txt is a plain text file at the root of a site (yoursite.com/robots.txt). A group starts with one or more User-agent lines that name crawlers, followed by Allow and Disallow lines for paths. A crawler named in a group follows the groups that name it and ignores the group for every crawler (User-agent: *).
To let it in
User-agent: cohere-ai
Allow: /To keep it out
User-agent: cohere-ai
Disallow: /Does your site let it in?
cohere-ai is a name for robots.txt rules only. The crawler check reads robots.txt for it and sends no request as it.