| Operator | OpenAI |
|---|---|
| Purpose | Indexing for ChatGPT search |
| robots.txt token | OAI-SearchBot |
| Respects robots.txt | Yes, per OpenAI documentation. |
| If you block it | Your pages are not shown as sources in ChatGPT search answers. |
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot)Version numbers inside user-agents change over time. Match on the token (OAI-SearchBot), never on the full string.
robots.txt examples
Block OAI-SearchBot
User-agent: OAI-SearchBot
Disallow: /Allow OAI-SearchBot, block a folder
User-agent: OAI-SearchBot
Disallow: /private/
Allow: /A group for a specific crawler replaces the User-agent: * group for that crawler — rules are not combined. Repeat any rules from * that should still apply.
How to verify OAI-SearchBot
OpenAI publishes the IP ranges used by OAI-SearchBot.
The user-agent alone proves nothing — it is the first thing scrapers copy. See how to verify crawlers.
Is OAI-SearchBot blocked on your site?
Your robots.txt may allow OAI-SearchBot while your CDN or firewall refuses it. Test the real response with the AI crawler checker.
Official documentation: platform.openai.com