Applebot: a search crawler run by Apple
Applebot is a search crawler. It reads pages for the index behind this product: Siri and Spotlight. Operator: Apple.
At a glance
- Operator
- Apple
- robots.txt token
Applebot- Purpose
- Search. Search crawlers read pages to build the index an assistant searches when it answers. A page they cannot read cannot be found or cited in those answers.
- What it feeds
- Siri and Spotlight
What blocking it changes
If you keep Applebot out, your pages cannot be found and cited in the search answers of this product: Siri and Spotlight.
User agent
The full user agent, from the operator’s documentation. A request from Applebot carries this text, and a server log shows it.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)robots.txt lines for Applebot
robots.txt is a plain text file at the root of a site (yoursite.com/robots.txt). A group starts with one or more User-agent lines that name crawlers, followed by Allow and Disallow lines for paths. A crawler named in a group follows the groups that name it and ignores the group for every crawler (User-agent: *).
To let it in
User-agent: Applebot
Allow: /To keep it out
User-agent: Applebot
Disallow: /Does your site let it in?
The free crawler check reads your robots.txt for Applebot and requests your page with its user agent, so it shows whether a robots.txt rule or a firewall turns it away.