AI crawlers
Bingbot: a search crawler run by Microsoft
Bingbot is a search crawler. It reads pages for the index behind this product: Bing and Copilot. Operator: Microsoft.
At a glance
- Operator
- Microsoft
- robots.txt token
Bingbot- Purpose
- Search. Search crawlers read pages to build the index an assistant searches when it answers. A page they cannot read cannot be found or cited in those answers.
- What it feeds
- Bing and Copilot
What blocking it changes
If you keep Bingbot out, your pages cannot be found and cited in the search answers of this product: Bing and Copilot.
robots.txt lines for Bingbot
robots.txt is a plain text file at the root of a site (yoursite.com/robots.txt). A group starts with one or more User-agent lines that name crawlers, followed by Allow and Disallow lines for paths. A crawler named in a group follows the groups that name it and ignores the group for every crawler (User-agent: *).
To let it in
User-agent: Bingbot
Allow: /To keep it out
User-agent: Bingbot
Disallow: /Does your site let it in?
The crawler check reads only robots.txt for Bingbot and sends no request as it.