Applebot-Extended: a robots.txt control for Apple

Applebot-Extended is a robots.txt control for using Applebot data to train generative models. It sends no separate HTTP request.

At a glance

Operator
Apple
robots.txt token
Applebot-Extended
Purpose
Training. Training crawlers collect pages for future models. Keeping them out does not stop the crawlers that fetch pages for answers, which have names of their own.
What it feeds
Apple Intelligence training

What blocking it changes

If you keep Applebot-Extended out, your pages are left out of what it collects for this: Apple Intelligence training. The crawlers that fetch pages for answers have names of their own, so they can still read your site.

robots.txt lines for Applebot-Extended

robots.txt is a plain text file at the root of a site (yoursite.com/robots.txt). A group starts with one or more User-agent lines that name crawlers, followed by Allow and Disallow lines for paths. A crawler named in a group follows the groups that name it and ignores the group for every crawler (User-agent: *).

To let it in

User-agent: Applebot-Extended
Allow: /

To keep it out

User-agent: Applebot-Extended
Disallow: /

Does your site let it in?

Applebot-Extended is a name for robots.txt rules only. The crawler check reads robots.txt for it and sends no request as it.

Check whether your site lets Applebot-Extended in

Questions about Applebot-Extended

Applebot-Extended is a robots.txt control for using Applebot data to train generative models. It sends no separate HTTP request.

That is your choice. Blocking it keeps your pages out of what it collects for this: Apple Intelligence training. It does not block the crawlers that fetch pages for answers.

Add a group with its name and a rule for every path: “User-agent: Applebot-Extended” followed by “Disallow: /”. To let it in, write “Allow: /” instead.