AI crawler checker: can ChatGPT, Claude and Perplexity read your site?
Most AI crawler checkers only read robots.txt. That shows what you allow — not what actually happens. We fetch your page as each AI crawler and catch the blocks your CDN or firewall adds on top.
What this check covers
AI search crawlers
OAI-SearchBot, Claude-SearchBot, PerplexityBot — the ones that put you in AI answers with a link.
User-triggered agents
ChatGPT-User, Claude-User, Perplexity-User — fetches made when someone pastes your link.
Training crawlers
GPTBot, ClaudeBot, Google-Extended — blocking these does not hurt visibility.
robots.txt vs reality
Flags crawlers that robots.txt allows but the server refuses.
Cloudflare AI Crawl Control
Recognises the block pattern of Cloudflare's AI settings, including the Googlebot side effect.
llms.txt
Checks whether an llms.txt file is published.
Questions
Why can't ChatGPT see my website?
The common causes: robots.txt disallows OAI-SearchBot or ChatGPT-User, a CDN setting blocks AI crawlers (Cloudflare does this by default for many sites since July 2025), or the content is rendered only with JavaScript.
Should I block AI crawlers?
Block training crawlers if you do not want your content used to train models. Keep AI search crawlers and user agents allowed if you want to be cited in AI answers. They are different bots and can be controlled separately.
Does blocking GPTBot remove me from ChatGPT search?
No. ChatGPT search uses OAI-SearchBot. GPTBot is the training crawler. Blocking GPTBot alone does not affect search visibility.
Related guides
Block or allow AI crawlers? A robots.txt guide for 2026
Which AI bots train models and which put you in AI answers, ready-to-use robots.txt snippets for each policy, and why robots.txt alone is not enough.
Guide · 6 minCloudflare AI Crawl Control: which setting to pick (and which one removes you from Google)
Search, Agent and Training in Cloudflare AI Crawl Control: what each controls, Block vs Disallow AI Training, the September 2026 changes and safe defaults.
More free tools
Googlebot checker
Free Googlebot test: fetch any URL as Googlebot Smartphone and Desktop, see robots.txt rules, noindex, CDN blocks and what content Google receives.
robots.txt tester
Test your robots.txt online: see which group applies to Googlebot, Bingbot, GPTBot, ClaudeBot and other crawlers, and which rule matches your URL.
Cloudflare SEO checker
Check if Cloudflare blocks Googlebot, Bingbot or AI crawlers on your site. Detects Bot Fight Mode challenges and AI Crawl Control blocks; reads your settings with a read-only token.
Test from different countries
Check if your website loads from different locations and whether visitors or crawlers in some regions get blocked by a CDN, WAF or geo rule.