Cookies on AIGROW

We use necessary cookies to run the site. With your permission, we also use cookies for analytics, session recordings and marketing attribution.

Cookie Policy

AI crawler access setup: robots.txt, CDN rules and sitemap set so the AI crawlers you choose can read your site

$290Fixed price, paid onceReport within 5 working days of access

Order for $290How it works

The same checks before and after, and again 30 days later. Delivered as listed, or refunded in full.

AIGROW sets up AI crawler access on one website for $290 paid once. We check what the main AI search and training crawlers can reach through your robots.txt, CDN and firewall rules, redirects and HTML, set the rules as you choose, fix your sitemap, submit it to Bing and IndexNow, and check again 30 days later.

What the setup does, and what it does not

A site can look fine in a browser and still turn AI crawlers away: an old robots.txt rule copied from another site, a CDN or firewall setting for AI bots, text that only appears after JavaScript runs, or a broken sitemap. Some crawlers fetch pages for AI search and answers, others collect pages to train AI models, and you choose which ones to let in.

What it does
  • Checks what the main AI crawlers can reach on up to 10 pages: those that fetch pages for AI search and answers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, and the training crawlers, such as GPTBot, ClaudeBot and CCBot
  • Reads your robots.txt, your CDN or firewall rules for bots, the HTTP status and redirects of each page, and whether the main text is in the HTML before JavaScript runs
  • Writes your choice down on a decision sheet: which crawlers to allow for search and answers, and which training crawlers to block, after we explain the trade-off in plain words
  • Sets robots.txt and your CDN rule as you chose and fixes your XML sitemap, with the access you give us, or sends the files and exact steps to your developer
  • Submits your sitemap to Bing Webmaster Tools and IndexNow where you give us access, and adds an llms.txt file if you want one
  • Runs the same checks again after the changes and 30 days later
What it does not do
  • It does not decide for you which crawlers to allow: we explain, you choose
  • It does not promise that any AI assistant will cite, mention or recommend you, or any ranking
  • It does not rebuild your site: if the main text only appears after JavaScript, we report it and write down what your developer needs to change
  • It does not stop crawlers that ignore robots.txt: a rule states your preference, and some requests made for a user do not follow it
  • It does not need your password: you invite hello@aigrow.gg, or create a temporary user for it that is removed after the report

From payment to the recheck

Each step and when it happens. Nothing changes on your site before you confirm your choices in writing.

Pay at checkoutPay once by card at the Stripe checkout. Stripe sends the receipt.2 minutes
Answer and give us accessWe email you straight after payment. Reply with your site, your CDN or host and your pages, then invite hello@aigrow.gg or create a temporary user for it.15 minutes
We check what the crawlers reachFor every page and crawler we record the robots.txt rule, the CDN setting, the HTTP status and redirects, and whether the main text is in the HTML. Then we send you the decision sheet with the trade-off explained.Within 5 working daysFrom access to the report
We make the changes you choseWe set robots.txt, your CDN rule and your sitemap, add llms.txt if you asked for it, and submit the sitemap to Bing and IndexNow. Or we send the files and steps to your developer.After your written OK
Read the reportThe same checks before and after, page by page and crawler by crawler, and then we remove our access. 30 days later we run the checks again and send you the result.With the changesRecheck at 30 days

One website, its crawler rules, one price

Paid once at checkout, which asks for your email and card. Right after payment we email you a short questionnaire and how to give us access without a password; the report follows within 5 working days of access.

What we need from you

Reply to our questionnaire with your site, its platform, your CDN or host, up to 10 pages and your choices. Then invite hello@aigrow.gg to your CDN account, your site platform or host, and Bing Webmaster Tools, or create a temporary user for that address. Never send a password by email.

What you receive

  • A check of up to 10 pages for each main AI crawler: robots.txt, CDN and firewall rules, HTTP status and redirects, and the main text in the HTML
  • A decision sheet with your choice for search and answer crawlers and for training crawlers
  • robots.txt and your CDN rule set as you chose, or the files and exact steps for your developer
  • Your XML sitemap fixed, and submitted to Bing Webmaster Tools and IndexNow where you give access
  • An llms.txt file, if you want one
  • A report with the same checks before and after, within 5 working days of access
  • Our access removed after the report, and one recheck 30 days later

AI crawler access setup

$290

Paid once at checkout. No subscription, nothing renews.

If we do not deliver what this page lists, or you change your mind before work starts, you get a full refund.

Why buy it from AIGROW

What you can check before you pay, and after the work.

A fixed price
$290, paid once. No subscription, nothing renews. Everything included is listed on this page before you pay.
Proof you can check
The report shows every page and crawler before and after: the robots.txt rule that applies, the CDN setting, the HTTP status and whether the main text is in the HTML. You can run the same check yourself with the free AI crawler check.
A public method
The first check is one you can run yourself: the free AI crawler check reads your robots.txt and requests a page as each crawler, in a few seconds, with no account. Run the free AI crawler check

Before you buy

What we check and change, and what the price does not promise.

Nobody can promise that, and we do not. The setup makes sure your own rules do not turn away the crawlers you chose, and that your pages and sitemap can be read. Whether an assistant uses or cites a page is up to its operator. Bing’s webmaster guidelines say the same of this kind of work: it does not guarantee citations.

Some crawlers fetch pages so an AI assistant can find and quote them in an answer, such as OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot. Others collect pages to train AI models, such as GPTBot, ClaudeBot and CCBot. OpenAI and Anthropic document a separate control for each, so you can block training and still allow search. We explain the trade-off, and the choice is yours.

A CDN or firewall can block AI crawlers through a setting you may not know is on. Cloudflare lets each site allow or block AI crawlers by category, and domains added to it since 15 September 2026 block training and agent crawlers by default on pages that show ads. Vercel offers a firewall ruleset for AI bots. We read the setting that applies to your site.

llms.txt is a proposal from September 2024 for a file that points language models to the main pages of a site. No major AI assistant has confirmed that it reads the file, and Google says its Search ignores it. If you want one, we write it from your own pages; if you do not, we leave it out.

Bing says its Copilot search relies on the same crawling and indexing as Bing search, and that it finds pages through XML sitemaps and IndexNow. So where you give us access, we submit your sitemap in Bing Webmaster Tools and send your pages through IndexNow.

We report it page by page. Bing’s guidelines warn against hiding important content behind client-side rendering, and a study by Vercel and MERJ published in December 2024 found that the crawlers of OpenAI, Anthropic and Perplexity did not run JavaScript. Changing how your site renders is development work outside this setup, so we write down what your developer needs to change.

No, and never send one by email. You invite hello@aigrow.gg as a member of your CDN account limited to this site, as a user on your site platform or host, and as a user in Bing Webmaster Tools. Where a platform cannot invite, you create a temporary user for hello@aigrow.gg and the platform emails the sign-in link to us. We keep it in our password vault and remove the access once the report is sent.

The report with the checks before and after comes within 5 working days of access; the days we wait for your choices or your developer do not count. Crawlers pick up changes at their own pace, and OpenAI says a robots.txt change can take about 24 hours to reach its systems, so we run the same checks again 30 days later.

Want to know what the assistants already say about you?

The AI accuracy audit asks ChatGPT, Claude, Gemini and Grok what customers ask about your business, marks every wrong answer against your real details and writes the fix plan.

See the AI accuracy audit

The service that fits next

$390

Structured data and answer pages

We add schema.org markup to up to 20 pages of your site and write 5 pages answering customer questions.

See structured data and answer pages

Let the AI crawlers you choose read your site.

Pay once, give us access without a password, choose which crawlers to allow, and receive the report within 5 working days of access.

Order for $290See every done-for-you service

$290 · paid once · delivered as listed or refunded