AI Crawler Checker: can AI read your site?

Enter your website. In a few seconds you will see which AI crawlers from OpenAI, Anthropic, Google, Perplexity, Apple, Meta and Amazon are allowed to read it, and the exact rule that decides.

Checks the page you enter, so paste a specific page to test just that page. We read two public files, your robots.txt and that page, and store nothing.

Reading robots.txt and the page…

Three kinds of AI crawler, and why the difference matters

Most advice says “block AI bots” or “allow AI bots” as if they were one thing. They are not. Each company runs separate crawlers for separate jobs, and blocking the wrong one has a different cost.

Search crawlers

Build the index an assistant answers from. Block one and your site drops out of that assistant’s answers and citations.

On-request fetchers

Open a page because a person asked the assistant about it. Several companies say robots.txt may not apply to these visits.

Training crawlers

Collect text to train future models. Blocking them is a choice about your content. It does not remove you from live answers.

So a business can block GPTBot and still appear in ChatGPT, because ChatGPT search reads through OAI-SearchBot. The mistake we see most often is the reverse: a security plugin or a blanket Disallow: / that shuts out the search crawlers too. Our post on Cloudflare’s AI crawler controls covers how that happens without anyone deciding it.

The crawlers this tool checks

Names and purposes come from each company’s own crawler documentation, read on November 11, 2026. Each row links to that page. Microsoft’s crawler is not listed because its documentation page could not be read when this list was checked.

Company	robots.txt name	Type	What it does
OpenAI	`OAI-SearchBot`	Search	Surfaces sites in ChatGPT search results. Source
OpenAI	`ChatGPT-User`	On request	Visits a page when someone asks ChatGPT about it. OpenAI says robots.txt rules may not apply to these visits. Source
OpenAI	`GPTBot`	Training	Collects content that may be used to train OpenAI models. Source
Anthropic	`Claude-SearchBot`	Search	Indexes the web to improve Claude search results. Source
Anthropic	`Claude-User`	On request	Fetches a page when a Claude user asks about it. Source
Anthropic	`ClaudeBot`	Training	Collects content that could contribute to training Claude. Source
Google	`Googlebot`	Search	Crawls for Google Search and all of its features. Source
Google	`Google-Extended`	Training	A switch, not a crawler. Controls whether pages Google has crawled may be used to train Gemini models and to ground answers in the Gemini apps. Google says it does not affect Google Search. Source
Perplexity	`PerplexityBot`	Search	Surfaces and links sites in Perplexity search results. Source
Perplexity	`Perplexity-User`	On request	Fetches a page when a Perplexity user asks about it. Perplexity says this fetcher generally ignores robots.txt. Source
Apple	`Applebot`	Search	Powers search in Siri, Spotlight and Safari. With no rules of its own, it follows the rules written for Googlebot. Source
Apple	`Applebot-Extended`	Training	A switch, not a crawler. Opts pages out of training Apple foundation models. Source
Meta	`Meta-WebIndexer`	Search	Indexes pages so Meta AI can cite and link to them. Source
Meta	`Meta-ExternalFetcher`	On request	Fetches individual links at a user’s request. Meta says it may bypass robots.txt. Source
Meta	`Meta-ExternalAgent`	Training	Crawls for training Meta’s AI models or improving its products. Source
Amazon	`Amzn-SearchBot`	Search	Improves search in Amazon products such as Alexa. Amazon says it does not crawl for model training. Source
Amazon	`Amzn-User`	On request	Supports user actions such as answering an Alexa query. Amazon says it may not follow all robots.txt directives. Source
Amazon	`Amazonbot`	Training	Improves Amazon products and may be used to train Amazon AI models. Source
Common Crawl	`CCBot`	Training	Builds Common Crawl’s open repository of web crawl data, which anyone can download. Source

How robots.txt decides, in plain words

The rules a crawler must follow are written down in a public standard, RFC 9309. This tool applies them the same way:

A crawler obeys its own section first. If your file has rules addressed to it by name, it ignores the rules for * (everyone else).
The most specific rule wins. When an Allow and a Disallow both match a page, the longer path decides. On a tie, Allow wins.
No file means no restrictions. A missing robots.txt (404) lets every crawler in.
A broken file blocks everyone. If robots.txt returns a server error (5xx), crawlers must treat the whole site as off limits until it is fixed.

What this tool cannot see: a firewall or CDN setting that turns a crawler away before it reads robots.txt. Testing that would mean sending requests that pretend to be another company’s crawler, and those services check a crawler’s network address, not its name, so the result would be wrong. The tool identifies itself as Fade Digital and reports only what it can read.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. OpenAI uses GPTBot to collect content that may be used to train its models. ChatGPT search reads the web through a different crawler, OAI-SearchBot. If you block GPTBot and allow OAI-SearchBot, your pages can still appear in ChatGPT search results.

Should I block AI training crawlers?

That is a decision about your content, not about visibility. Blocking a training crawler asks that company not to use your pages to train future models. It does not change whether assistants cite you today, because citations come from the search crawlers. One thing to know first: Google-Extended also controls whether your pages can be used to ground answers in the Gemini apps, so blocking it costs more than training alone.

Do AI crawlers actually obey robots.txt?

OpenAI, Anthropic, Google, Apple and Amazon each state that their automated crawlers follow robots.txt. On-request fetchers are different: OpenAI, Perplexity, Meta and Amazon each say robots.txt may not apply when a page is fetched because a person asked for it. robots.txt is a request, not a lock. Only a firewall can enforce a block.

My robots.txt allows everything. Why might AI still not read my site?

Three common reasons. A firewall, CDN or security plugin can turn crawlers away before they ever read robots.txt. A noindex instruction on the page tells search engines not to list it. And a page that only shows its content after JavaScript runs can look empty to a crawler that does not run scripts. This tool reports the first two where it can see them.

How is this different from the free AI scan?

This tool checks one thing: whether AI crawlers have permission to read a page. The free AI scan goes further. It asks ChatGPT, Claude and Gemini the kind of questions your customers ask, reports whether your business is mentioned, and reviews the signals on your site that those answers draw on.

Being readable is the first step, not the result

A crawler that can read your site still has to find a reason to recommend you. The free AI scan asks ChatGPT, Claude and Gemini the questions your customers ask and shows whether your business comes up.

See how AI picks a business →