WHO IS ACTUALLY CRAWLING YOUR SITE
AI crawlers, explained
There is not one “AI crawler.” There are several, run by different companies, doing genuinely different jobs, and lumping them together is how sites end up blocking the wrong ones by accident. Here is what each one actually does, and how to check and control access yourself.
THE FIRST DISTINCTION THAT MATTERS
Training crawlers are not the same as retrieval crawlers
A training crawler visits your site to potentially include it in the data used to build a future AI model. A retrieval crawler visits your site right now, either to build a search index the AI can cite, or because a live user asked the assistant to go look at your page. Blocking a training crawler is a reasonable, common choice that has nothing to do with whether your business shows up in an AI answer today. Blocking a retrieval crawler is closer to blocking Googlebot: it can make you invisible to the exact tool people are using to find you.
BY COMPANY
What each crawler does
Crawls content that may be used to train future OpenAI models. Blocking it opts your site out of training data without affecting ChatGPT’s live search results. OpenAI’s own publisher documentation covers each of its crawlers individually.
OAI-SearchBot indexes pages so ChatGPT search can cite them. ChatGPT-User fetches a specific page in real time when a person asks ChatGPT to read it. Both are retrieval, not training, and can be allowed independently of GPTBot.
Collects web content that may contribute to training Anthropic’s models. Anthropic documents all three of its crawlers and how to block each one individually.
Claude-SearchBot indexes content to improve Claude’s search results. Claude-User fetches a page live when someone using Claude asks a question that requires reading it. Neither is a training crawler.
PerplexityBot indexes content for citations in Perplexity’s answers. Perplexity-User fetches pages live at a user’s request. Perplexity’s documentation lists both and how each respects robots.txt.
Controls whether your content can be used to improve Gemini and Vertex AI. It has no effect on your ranking or inclusion in regular Google Search, which is handled entirely by Googlebot. Google’s own crawler documentation explains the distinction.
A separate opt-out specifically for Apple’s generative AI features, distinct from the regular Applebot crawler that powers Siri and Spotlight search results. Apple’s support documentation covers the difference.
HOW TO CHECK
Verify your own site instead of guessing
Robots.txt is only half the picture. A security plugin or a CDN’s bot-protection setting can block these crawlers at the server level even when robots.txt allows them, and almost nobody checks for that directly. The free health check includes exactly this check, whether AI crawlers can actually reach your site in practice, and flags separately if you are blocking only the training crawlers, which is often a deliberate and sensible choice rather than a mistake.
This is a two-minute fix, once you know what to look for
Most sites that block AI crawlers did not decide to. A security plugin or CDN shipped an aggressive bot-blocking default, and nobody went back to check who got caught in it. Run the free check first, it will tell you exactly which crawlers can and cannot reach you right now.
Bill
Next step: run the free check to see which crawlers can reach your site right now. For the llms.txt file these systems sometimes read alongside crawling, see what is llms.txt. For the bigger picture, see what is AI visibility.