Cloudflare’s AI Crawl Control groups crawlers by purpose and lets you choose an action per purpose. The labels look similar, but the consequences are very different — one of them can take your site out of Google.
The three purposes
| Purpose | Examples | What blocking does |
|---|---|---|
| Search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | You stop being cited (with links) in AI answers. |
| Agent | ChatGPT-User, Claude-User, Perplexity-User | AI assistants can’t open your pages when a user asks them to. |
| Training | GPTBot, ClaudeBot — and mixed-use crawlers like Googlebot | With “Block”: training crawlers and Googlebot, Bingbot, Applebot are refused. |
Block vs Disallow AI Training
Block refuses the request at Cloudflare’s edge. Since 15 September 2026, it applies to every crawler that has training among its purposes — Googlebot, Bingbot and Applebot included.
Disallow AI Training publishes a no-training preference in your robots.txt (Bot Preference Sync), keeps accountable mixed-use crawlers allowed for search, and blocks training-only crawlers such as GPTBot and ClaudeBot. It is available for the Training category only and does not apply to pages with ads.
If you want out of AI training but want to stay in Google: choose Disallow AI Training, never Block.
What happened to old settings
According to Cloudflare, legacy “Block AI bots” values were migrated to Disallow AI Training for Training, with Agent set to block on pages with ads and Search left allowed. “Block AI Bots” and Managed robots.txt are deprecated. If someone later switches Training to Block, the Googlebot side effect applies.
New domain presets
- Sites without ads: Search, Training and Agent all Allow.
- Ad-supported sites: Search Allow, Training “Disallow AI Training”, Agent blocked on pages with ads.
A safe configuration for most businesses
- Search: Allow — be citable in ChatGPT, Claude and Perplexity answers.
- Agent: Allow — let assistants open your pages for users.
- Training: Allow or Disallow AI Training — your policy choice.
Verify the result
Settings are only half the story — WAF rules and Bot Fight Mode act independently. Run the AI crawler checker to see the response each crawler actually gets.