OpenAI runs three separate user-agents. They can be allowed or blocked independently in robots.txt, and confusing them is the most common robots.txt mistake we see.
| GPTBot | OAI-SearchBot | ChatGPT-User | |
|---|---|---|---|
| Purpose | Model training | ChatGPT search index | Fetches on a user’s request |
| Block it and… | Content excluded from training | Not cited in ChatGPT search | ChatGPT can’t open your links |
| Affects visibility? | No | Yes | Yes (for users sharing your links) |
| robots.txt token | GPTBot | OAI-SearchBot | ChatGPT-User |
Most common setup
Block GPTBot, allow OAI-SearchBot and ChatGPT-User: no training, full visibility. See the robots.txt snippets.
Check what your CDN does
robots.txt can say “allow” while the CDN refuses the request. Test all three with the AI crawler checker.