Can AI read my website?

Can AI read my website? Compare raw HTML text, headings, links and structured data across crawler requests. Check access rules and spot JavaScript shells.

Free, no account. Ten checks per hour across the AI readability, mobile and technology checkers.

We send the address to our API for this check. Results can be cached for five minutes.

AI crawlers read the HTML your server sends. In Vercel’s December 2024 study, the crawlers behind ChatGPT, Claude and Perplexity did not run JavaScript, so words added by scripts never reached them. Enter a page to fetch it as four AI crawlers and a browser, then see the text, headings and links each one received.

How to use this checker

Five requests for one page
Our server asks for the page as GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, using the user agent each operator publishes, and once as a desktop Chrome browser. Each request follows up to three redirects and reads at most 3 MB.
robots.txt comes first
We read the site’s robots.txt before any crawler request. A crawler that robots.txt turns away is never sent to the page, and the rule that blocks it is shown.
Page directives and llms.txt
Each crawler row shows restrictive meta robots and X-Robots-Tag directives, with the operator’s user agent documentation. noai and noimageai are non-standard requests. llms.txt is checked as a minor voluntary signal. The rendered text share is not measured; word-count comparisons use desktop raw HTML.
No JavaScript runs
Text, headings, links, title, description and JSON-LD come from the HTML exactly as the server sent it. Nothing on the page is executed. That matches how the crawlers in Vercel’s study read pages.
How the verdicts are given
Readable: 40 or more words arrived and nothing suggests the rest needs JavaScript. Partly readable: a short page, under 150 words with signs of a script-built page, or less than half the words the browser copy holds. Empty shell: no readable words with scripts, or under 40 words with an empty app root, a JavaScript notice or heavy inline scripts. Blocked: robots.txt, a refusal or a bot challenge stopped the crawler.

Source for what crawlers run: Vercel, “The rise of the AI crawler”, December 17, 2024. The crawlers it measured from OpenAI, Anthropic, Meta, ByteDance and Perplexity did not run JavaScript.

Questions about this checker

In Vercel’s study “The rise of the AI crawler” (December 17, 2024), none of the major AI crawlers rendered JavaScript, including OpenAI’s OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic’s ClaudeBot and PerplexityBot. ChatGPT and Claude fetched JavaScript files without running them. Gemini uses Googlebot’s infrastructure, which renders JavaScript. Crawlers can change, so test again after a redesign.

Your browser runs the page’s scripts, which build the text after loading. A crawler that runs no scripts keeps the first HTML, and in a client-rendered app that is an empty container. Server rendering, static generation or prerendering puts the finished text into that first HTML.

Some sites send crawlers a prerendered copy and browsers the script-built page. The check flags this when a crawler receives at least 150 words and the browser copy fewer than 40. Keep both copies the same in substance, since search engines treat different content for bots and people as cloaking.

A firewall or bot protection service can refuse a request or answer with a challenge page before the site sees it. Allow the crawler in that service, for example in Cloudflare’s bot settings. Our requests come from one server, so a block here is a strong sign, though not proof, of what the real crawler gets.

No. It only shows what a crawler can read on one page. Whether an assistant names your business depends on many other sources. The free AI visibility scan checks real answers.

Related tools

  • Website technology checker

    Use this website technology checker to identify software from visible markers and DNS. Inspect disclosed versions and evidence, then compare two sites with AI crawler access notes.

  • AI crawler check

    Use this AI crawler checker to review robots.txt rules, indexing signals and HTTP responses. Inspect blocked access before changing your site's rules.

  • llms.txt generator and validator

    Use this llms.txt generator to build an editable file from a website, XML sitemap or pasted page list. Organize sections, review descriptions, then copy or download the Markdown.

Check your business in AI answers

A page that crawlers can read is the starting point. The free AI visibility scan shows how assistants answer about your business and what to work on next.

Run the free AI visibility scan