Guides
How crawlers reach your site, what blocks them, and how to fix it without switching off protection.
Cloudflare blocking Googlebot: how to find the setting and fix it
Why Cloudflare can return 403 to Googlebot — AI training blocks, Bot Fight Mode, WAF rules — and how to confirm and fix it without hurting your bot protection.
Guide · 6 minCloudflare AI Crawl Control: which setting to pick (and which one removes you from Google)
Search, Agent and Training in Cloudflare AI Crawl Control: what each controls, Block vs Disallow AI Training, the September 2026 changes and safe defaults.
Guide · 5 minDoes Cloudflare Bot Fight Mode hurt SEO?
What Bot Fight Mode challenges, why it is hard to whitelist on the Free plan, how it affects crawlers and monitoring, and when to use Super Bot Fight Mode instead.
Guide · 6 min“Blocked due to access forbidden (403)” in Google Search Console: causes and fixes
What the “Blocked due to access forbidden (403)” status means, the most common causes (CDN, WAF, security plugins, login walls) and a step-by-step fix.
Guide · 5 minHow to verify Googlebot (and spot fake Googlebots)
Verify that a request claiming to be Googlebot really comes from Google using reverse DNS or Google’s published IP ranges. Includes commands and common mistakes.
Guide · 7 minBlock or allow AI crawlers? A robots.txt guide for 2026
Which AI bots train models and which put you in AI answers, ready-to-use robots.txt snippets for each policy, and why robots.txt alone is not enough.
Comparisons
OAI-SearchBot vs GPTBot (and ChatGPT-User)
The three OpenAI user-agents compared: which one trains models, which one powers ChatGPT search, which one fetches pages for users — and what blocking each does.
CompareGPTBot vs ChatGPT-User
GPTBot crawls for training; ChatGPT-User fetches a page when a person asks ChatGPT about it. Differences, robots.txt behaviour and when to block each.
CompareCloudflare Bot Fight Mode vs Super Bot Fight Mode
Cloudflare’s free Bot Fight Mode vs paid Super Bot Fight Mode: plans, control, whitelisting and what each means for search and AI crawlers.
CompareCrawlability vs indexability
Crawlability is whether a crawler can fetch a page; indexability is whether it may be added to the index. How robots.txt, noindex, 403s and canonicals affect each.