Home / Free tools

robots.txt tester for Googlebot and AI crawlers

Google retired the robots.txt tester in Search Console. This one tests your live file against 14 crawlers at once, using the same matching rules Google uses: most specific user-agent group, longest path match, allow wins ties.

Free · No sign-up · About 10 seconds

What this check covers

Per-crawler evaluation

Which user-agent group each crawler falls into and the matching rule.

Wildcards and $

Path patterns evaluated like Google's open-source parser.

Fetch status

5xx or a timeout on robots.txt means Google stops crawling — we flag it.

Challenge pages

Detects when the CDN serves a challenge instead of the file.

Sitemap lines

Lists declared sitemaps and fetches the first one.

Managed robots.txt

Notices content added by your CDN on top of your own file.

Questions

Where is the robots.txt tester in Google Search Console?

Google removed it and replaced it with a robots.txt report (Settings → robots.txt) that shows fetch status but does not test individual URLs.

What happens if robots.txt returns 404?

Crawlers treat a missing robots.txt as "everything allowed". A 5xx error is different: Google treats the site as fully disallowed until the file is reachable again.

Does robots.txt stop a page from being indexed?

No. It stops crawling. A blocked URL can still be indexed from links without content. Use noindex to keep a page out of results — and do not block it in robots.txt, or Google never sees the noindex.

Related guides

Guide · 7 min

Block or allow AI crawlers? A robots.txt guide for 2026

Which AI bots train models and which put you in AI answers, ready-to-use robots.txt snippets for each policy, and why robots.txt alone is not enough.

Guide · 6 min

“Blocked due to access forbidden (403)” in Google Search Console: causes and fixes

What the “Blocked due to access forbidden (403)” status means, the most common causes (CDN, WAF, security plugins, login walls) and a step-by-step fix.