ClaudeBot user agent, robots.txt & how to verify

Operator	Anthropic
Purpose	Collecting data to train Anthropic models
robots.txt token	`ClaudeBot`
Respects robots.txt	Yes, per Anthropic documentation.
If you block it	Excludes your content from training. No effect on Claude search citations.

User-agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Version numbers inside user-agents change over time. Match on the token (ClaudeBot), never on the full string.

robots.txt examples

Block ClaudeBot

User-agent: ClaudeBot
Disallow: /

Allow ClaudeBot, block a folder

User-agent: ClaudeBot
Disallow: /private/
Allow: /

A group for a specific crawler replaces the User-agent: * group for that crawler — rules are not combined. Repeat any rules from * that should still apply.

How to verify ClaudeBot

Anthropic publishes the IP ranges used by its crawlers.

The user-agent alone proves nothing — it is the first thing scrapers copy. See how to verify crawlers.

Is ClaudeBot blocked on your site?

Your robots.txt may allow ClaudeBot while your CDN or firewall refuses it. Test the real response with the AI crawler checker.

Official documentation: support.anthropic.com