Meta-ExternalFetcher: a crawler run by Meta that opens pages on request
Meta-ExternalFetcher is a user crawler. It opens a page when a person asks this product to read it: Meta AI. Operator: Meta.
At a glance
- Operator
- Meta
- robots.txt token
Meta-ExternalFetcher- Purpose
- User request. User crawlers open one page when a person asks an assistant to read it. A page they cannot read cannot be opened in that conversation.
- What it feeds
- Meta AI
What blocking it changes
A Disallow asks Meta-ExternalFetcher to stay out of your pages. Some user-triggered fetchers may ignore robots.txt; use server access controls if a page must remain private.
User agent
A representative user agent for Meta-ExternalFetcher. We use operator documentation where a full string is published. Versions and browser details in real server logs can differ.
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)robots.txt lines for Meta-ExternalFetcher
robots.txt is a plain text file at the root of a site (yoursite.com/robots.txt). A group starts with one or more User-agent lines that name crawlers, followed by Allow and Disallow lines for paths. A crawler named in a group follows the groups that name it and ignores the group for every crawler (User-agent: *).
To let it in
User-agent: Meta-ExternalFetcher
Allow: /To keep it out
User-agent: Meta-ExternalFetcher
Disallow: /Does your site let it in?
The free crawler check reads your robots.txt for Meta-ExternalFetcher and requests your page with its user agent, so it shows whether a robots.txt rule or a firewall turns it away.