Googlebot: a search crawler run by Google

Googlebot is a search crawler. It reads pages for the index behind this product: Google Search and AI Overviews. Operator: Google.

At a glance

Operator
Google
robots.txt token
Googlebot
Purpose
Search. Search crawlers read pages to build the index an assistant searches when it answers. A page they cannot read cannot be found or cited in those answers.
What it feeds
Google Search and AI Overviews

What blocking it changes

If you keep Googlebot out, your pages cannot be found and cited in the search answers of this product: Google Search and AI Overviews.

robots.txt lines for Googlebot

robots.txt is a plain text file at the root of a site (yoursite.com/robots.txt). A group starts with one or more User-agent lines that name crawlers, followed by Allow and Disallow lines for paths. A crawler named in a group follows the groups that name it and ignores the group for every crawler (User-agent: *).

To let it in

User-agent: Googlebot
Allow: /

To keep it out

User-agent: Googlebot
Disallow: /

Does your site let it in?

The crawler check reads only robots.txt for Googlebot and sends no request as it.

Check whether your site lets Googlebot in

Questions about Googlebot

Googlebot is a search crawler. It reads pages for the index behind this product: Google Search and AI Overviews. Operator: Google.

Only if you do not want your pages in the search answers of this product: Google Search and AI Overviews. Blocking it changes nothing for crawlers with other names.

Add a group with its name and a rule for every path: “User-agent: Googlebot” followed by “Disallow: /”. To let it in, write “Allow: /” instead.