Meta-WebIndexer: a search crawler run by Meta

Meta-WebIndexer is a search crawler. It reads pages for the index behind this product: Meta AI search. Operator: Meta.

At a glance

Operator
Meta
robots.txt token
Meta-WebIndexer
Purpose
Search. Search crawlers read pages to build the index an assistant searches when it answers. A page they cannot read cannot be found or cited in those answers.
What it feeds
Meta AI search

What blocking it changes

If you keep Meta-WebIndexer out, your pages cannot be found and cited in the search answers of this product: Meta AI search.

User agent

A representative user agent for Meta-WebIndexer. We use operator documentation where a full string is published. Versions and browser details in real server logs can differ.

meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)

robots.txt lines for Meta-WebIndexer

robots.txt is a plain text file at the root of a site (yoursite.com/robots.txt). A group starts with one or more User-agent lines that name crawlers, followed by Allow and Disallow lines for paths. A crawler named in a group follows the groups that name it and ignores the group for every crawler (User-agent: *).

To let it in

User-agent: Meta-WebIndexer
Allow: /

To keep it out

User-agent: Meta-WebIndexer
Disallow: /

Does your site let it in?

The free crawler check reads your robots.txt for Meta-WebIndexer and requests your page with its user agent, so it shows whether a robots.txt rule or a firewall turns it away.

Check whether your site lets Meta-WebIndexer in

Questions about Meta-WebIndexer

Meta-WebIndexer is a search crawler. It reads pages for the index behind this product: Meta AI search. Operator: Meta.

Only if you do not want your pages in the search answers of this product: Meta AI search. Blocking it changes nothing for crawlers with other names.

Add a group with its name and a rule for every path: “User-agent: Meta-WebIndexer” followed by “Disallow: /”. To let it in, write “Allow: /” instead.