AI Crawlability Checker
See which AI crawlers can reach your site, and which ones are turned away.
| Bot | robots.txt | Crawlability |
|---|
robots.txt as fetched
What this checks
Two separate things decide whether an AI crawler reaches your page, and they fail independently.
The first is robots.txt, the polite gatekeeper. It is a request rather than a lock, and not every agent here honours it. The second is what your server does when the request actually arrives. That is the firewall layer, and it is where most accidental blocks live, because almost nobody sets them on purpose. A rule written to stop scrapers cannot tell a scraper apart from PerplexityBot.
This checker looks at both. It reads your robots.txt and works out what each bot is permitted to do, then fetches the page with that bot's real User-Agent and compares the response against what an ordinary browser gets. When the two layers disagree, you find out now instead of months later when you notice nothing is citing you.
One thing to be straight about. The probe runs from our servers using each bot's User-Agent string. Real crawlers arrive from their own IP ranges, so a firewall that verifies IP addresses can treat them differently. If you see a block here, it is real. An allowed result is a strong signal rather than a guarantee, and bots that are commonly verified by IP are marked as such in the results.
The AI crawlers we check
Not every user agent here does the same job. Some collect pages for training. Some build the index behind an assistant's search. Others only appear when a person asks the assistant to open a link. Blocking one is a different decision from blocking another, and the results are not interchangeable.
| User agent | Operator | What it does | If you block it |
|---|---|---|---|
GPTBot | OpenAI | Collects pages for model training | Your content is left out of training |
OAI-SearchBot | OpenAI | Builds the index behind ChatGPT search | ChatGPT search stops surfacing you |
ChatGPT-User | OpenAI | Fetches a page when someone asks ChatGPT to open it | ChatGPT cannot read a link a user pasted |
ClaudeBot | Anthropic | Collects pages for model training | Your content is left out of training |
Claude-SearchBot | Anthropic | Indexes pages for Claude's search results | Claude stops citing you in answers |
Claude-User | Anthropic | Fetches a page when someone asks Claude to open it | Claude cannot read a link a user pasted |
PerplexityBot | Perplexity | Indexes pages for Perplexity answers | Perplexity stops citing you |
Perplexity-User | Perplexity | Fetches a page in response to a question | Perplexity cannot open a page a user asked about |
Googlebot | The search index, which also feeds AI Overviews | You disappear from Google search | |
Bingbot | Microsoft | The Bing index, which feeds Copilot | You disappear from Bing and from Copilot |
Applebot | Apple | Powers search across Siri, Spotlight and Safari | Apple's assistants lose you |
Meta-ExternalAgent | Meta | Collects pages for Meta's models, and indexes some content directly | Your content is left out of Meta's training |
Meta-WebIndexer | Meta | Indexes pages for Meta AI's search results | Meta AI stops citing and linking to you |
DuckAssistBot | DuckDuckGo | Feeds DuckDuckGo's assistant answers, and is not used for training | DuckAssist stops using your pages |
MistralAI-Training | Mistral | Collects pages for model training | Your content is left out of Mistral's training |
MistralAI-Index | Mistral | Indexes pages for the search behind Mistral's assistant | Mistral's assistant stops surfacing you |
MistralAI-User | Mistral | Fetches a page when someone asks Mistral's assistant to open it | The assistant cannot read a link a user pasted |
CCBot | Common Crawl | Builds the open crawl archive many models train on | You are left out of a dataset used across the industry |
Google-Extended | A robots.txt directive, not a crawler. Decides whether Google may use your pages for Gemini | Your pages stay in Google search and in AI Overviews, but Google may not use them to train or ground Gemini | |
Applebot-Extended | Apple | A robots.txt directive, not a crawler. Decides whether Apple may train on your pages | Siri, Spotlight and Safari keep using your pages, Apple stops training its models on them |
Training and search are separate decisions
Blocking GPTBot while allowing OAI-SearchBot keeps you out of training data and keeps you in ChatGPT's search results. Anthropic splits ClaudeBot and Claude-SearchBot the same way, Meta splits Meta-ExternalAgent and Meta-WebIndexer, and Mistral runs a separate agent for each of training, search and user requests. Plenty of sites end up blocking everything by accident when all they wanted was to opt out of training.
The agents ending in User are not crawlers
ChatGPT-User, Claude-User, Perplexity-User and MistralAI-User only fire when somebody has already pasted your link and asked the assistant to read it. Blocking them protects nothing from training. It stops nothing except the person who was trying to read your page.
Whether blocking them in robots.txt even works depends on the operator. Anthropic and Mistral honour the file for these agents. OpenAI says the rules may not apply, because the request came from a person rather than from a crawl. Perplexity says its fetcher generally ignores them. A block that has to hold belongs in the firewall.
The last two are directives, not crawlers
Google-Extended and Applebot-Extended never send a request. They sit in robots.txt and tell Google and Apple what may be done with pages their ordinary crawlers already fetched. There is nothing to fetch them with, so the checker reports them from robots.txt only and marks them as having no live fetch.
Google-Extended is worth reading twice. Blocking it does not touch your Google rankings and does not remove you from AI Overviews, because both of those run on Googlebot. What it stops is Gemini using your content.
Apple falls back to Googlebot
If your robots.txt never names Applebot but does name Googlebot, Apple follows the Googlebot rules. The report says so on that row when it happens. It is worth knowing, because a Disallow you wrote with Google in mind quietly applies to Siri, Spotlight and Safari as well, and nothing in your file says that out loud.
A user agent is a claim, not an identity
Most operators here say they respect robots.txt, with the exceptions noted above, and anything can claim anything in a User-Agent string. If you need certainty rather than cooperation, verify by IP range or reverse DNS. The major operators publish theirs.
Why robots.txt is only half the answer
Reading robots.txt tells you what you asked for. It never shows you what actually happened.
Bot protection on a CDN, a rate limiter, a country block or a challenge page can all turn away an AI crawler while your robots.txt is still inviting it in. None of that is written in the file, and most of the time none of it reaches your logs either, because the request is cut off before it reaches your site.
The pattern is easy to recognise once you have seen it. Your robots.txt allows GPTBot, and GPTBot gets a 403 anyway. When robots.txt and reality disagree, the 403 is what is real.
This is also why a page can be missing from an assistant's answers while every technical check on the site comes back clean.
What each result means
The results table has two columns and they answer different questions, so the legend is split to match. Reading a Crawlability label against the robots.txt scale is the fastest way to misread the whole report.
The robots.txt column: what your file permits
| Result | What it means |
|---|---|
| Allowed | Your robots.txt lets this bot crawl the page you tested |
| Blocked | A rule in robots.txt closes this page to this bot. The row shows which rule did it, and the fix is in the file |
| Partly blocked | The page you tested is open to this bot, but robots.txt has rules aimed at it elsewhere on the site. The row shows which rules, and whose group they came from, so you can check you meant to keep them closed |
The Crawlability column: what your server actually did
| Result | What it means |
|---|---|
| Crawlable (with the HTTP status) | The bot's request went through and your server answered normally |
| Blocked (with the HTTP status) | The request was refused. robots.txt is not the cause, so look at your firewall, WAF or CDN rules |
| Needs a look | A bot that is normally verified by IP address was refused. Our probe does not come from that bot's own IP range, so this can be your firewall working correctly rather than a problem |
| Error (with the HTTP status) | The page returned an error rather than a block. A 404 usually means the URL is wrong, a 500 means something on your site is failing. Neither is a crawler issue |
| No response | Nothing came back at all. A timeout or a dropped connection, which points at the network or at a very aggressive block |
| No live fetch | A robots.txt directive rather than a crawler, so there is nothing to send a request to. The robots.txt verdict is the whole answer |
| Not tested | You did not select this bot, so no live request was made. Its robots.txt verdict still stands |
How to unblock a crawler
A rule in robots.txt. Remove the Disallow, or add a more specific Allow for that user agent. The most specific matching rule wins, so an explicit Allow for one bot beats a broad Disallow: / aimed at everyone.
A firewall or CDN rule. Nothing in your site's files will fix this one. On Cloudflare it lives in bot management. On other hosts look under WAF or bot protection. Allow the user agent there, and verify by IP range if you want to be careful about impostors.
A challenge page. Exempt the user agent from the challenge. A crawler cannot solve one.
A noindex in a meta tag or an X-Robots-Tag header. The crawler can reach the page and has been told not to keep it. Worth confirming that was deliberate, because a plugin or a leftover staging setting will do this quietly.
The same steps work in reverse. If you want to block GPTBot or any other AI crawler, a Disallow for that user agent in robots.txt is the polite way, and a firewall rule is the one that actually enforces it.
Frequently asked questions
What is AI crawlability?
Whether AI crawlers can reach and read your pages. It covers robots.txt and everything else that sits between a bot and your content, and it is the first requirement for anything else. LLM crawlability is the same idea under a different name.
Do AI crawlers respect robots.txt?
For the crawlers that fetch pages automatically, the major operators say they do. The agents that fetch a page because a person asked are a different case, and OpenAI and Perplexity both say robots.txt may not stop those. Nothing enforces any of it either way. robots.txt is a request, not a lock.
Can I block AI training but stay in AI search results?
Yes, and this is the option most site owners actually want. Block GPTBot and ClaudeBot to stay out of training, and allow OAI-SearchBot and Claude-SearchBot so the assistants can still find and cite you.
Does blocking AI crawlers hurt my SEO?
No. GPTBot, ClaudeBot and the other AI crawlers have nothing to do with Google rankings, and blocking them does not affect your position in search. Blocking Googlebot is a completely different matter.
Why does my site pass robots.txt checks and still get blocked?
Because robots.txt is only the first layer. A firewall, WAF or CDN rule can refuse the request before it reaches your server, and that block is invisible in both your robots.txt and your logs.
Is GPTBot the same as OAI-SearchBot?
No. GPTBot collects pages for training. OAI-SearchBot builds the index behind ChatGPT search. You can allow one and block the other.
Why doesn't ChatGPT cite my website or brand?
Being reachable is the first requirement, so start by checking that OAI-SearchBot is not blocked. If access is fine, the reason is usually that ChatGPT is finding the answer somewhere else and has no reason to reach for you.
How can I make my website show up in ChatGPT?
Make sure the crawlers can read you, then work on being worth citing. Assistants lean on sources they have seen quoted elsewhere, so mentions on sites they already trust matter more than anything on your own pages.
Being reachable is the first question. Whether the assistants actually name you is a different one.