| Operator | Anthropic |
|---|---|
| Purpose | Indexing to improve Claude search results |
| robots.txt token | Claude-SearchBot |
| Respects robots.txt | Yes, per Anthropic documentation. |
| If you block it | Claude is less likely to cite your pages in search answers. |
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com)Version numbers inside user-agents change over time. Match on the token (Claude-SearchBot), never on the full string.
robots.txt examples
Block Claude-SearchBot
User-agent: Claude-SearchBot
Disallow: /Allow Claude-SearchBot, block a folder
User-agent: Claude-SearchBot
Disallow: /private/
Allow: /A group for a specific crawler replaces the User-agent: * group for that crawler — rules are not combined. Repeat any rules from * that should still apply.
How to verify Claude-SearchBot
Anthropic publishes the IP ranges used by its crawlers.
The user-agent alone proves nothing — it is the first thing scrapers copy. See how to verify crawlers.
Is Claude-SearchBot blocked on your site?
Your robots.txt may allow Claude-SearchBot while your CDN or firewall refuses it. Test the real response with the AI crawler checker.
Official documentation: support.anthropic.com