Crawler directory
The crawlers that decide whether you show up in search and AI answers — what they do, how they identify themselves and what blocking them costs you.
| Crawler | Operator | Purpose | robots.txt token |
|---|---|---|---|
| Googlebot | Search indexing (also classified as AI training by Cloudflare) | Googlebot | |
| GPTBot | OpenAI | Collecting data to train OpenAI models | GPTBot |
| OAI-SearchBot | OpenAI | Indexing for ChatGPT search | OAI-SearchBot |
| ChatGPT-User | OpenAI | Fetching pages when a ChatGPT user asks for it | ChatGPT-User |
| ClaudeBot | Anthropic | Collecting data to train Anthropic models | ClaudeBot |
| Claude-SearchBot | Anthropic | Indexing to improve Claude search results | Claude-SearchBot |
| Claude-User | Anthropic | Fetching pages when a Claude user asks for it | Claude-User |
| PerplexityBot | Perplexity | Indexing for Perplexity search answers | PerplexityBot |
| Perplexity-User | Perplexity | Fetching pages when a Perplexity user asks for it | Perplexity-User |
| Bingbot | Microsoft | Bing search indexing (also feeds Copilot and many AI search products) | bingbot |
| Applebot | Apple | Search for Siri, Spotlight and Safari suggestions | Applebot |
| Google-Extended | robots.txt token controlling use for Gemini training and grounding | Google-Extended |