| Operator | Anthropic |
|---|---|
| Purpose | Fetching pages when a Claude user asks for it |
| robots.txt token | Claude-User |
| Respects robots.txt | Per Anthropic documentation; check the current policy for user-initiated fetches. |
| If you block it | Claude cannot read your page when a user asks it to. |
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)Version numbers inside user-agents change over time. Match on the token (Claude-User), never on the full string.
robots.txt examples
Block Claude-User
User-agent: Claude-User
Disallow: /Allow Claude-User, block a folder
User-agent: Claude-User
Disallow: /private/
Allow: /A group for a specific crawler replaces the User-agent: * group for that crawler — rules are not combined. Repeat any rules from * that should still apply.
How to verify Claude-User
Anthropic publishes the IP ranges used by its crawlers.
The user-agent alone proves nothing — it is the first thing scrapers copy. See how to verify crawlers.
Is Claude-User blocked on your site?
Your robots.txt may allow Claude-User while your CDN or firewall refuses it. Test the real response with the AI crawler checker.
Official documentation: support.anthropic.com