What OAI-SearchBot does
Crawls pages so they can appear as results and citations in ChatGPT search.
Allow it if you want ChatGPT search to cite your pages, even if you block GPTBot for training.
How to block OAI-SearchBot in robots.txt
Add these two lines to the robots.txt file at the root of your domain: User-agent: OAI-SearchBot and, on the next line, Disallow: /
To block only part of the site, replace / with a path such as /private/. Rules apply per host, so a subdomain needs its own robots.txt.
How to allow OAI-SearchBot
If you have no rule for OAI-SearchBot and no blanket User-agent: * Disallow: /, it is already allowed. To allow it explicitly while blocking other bots, add User-agent: OAI-SearchBot followed by Allow: /
Check it worked
- Publish the updated robots.txt and open it in a browser at /robots.txt to confirm the live file has the rule.
- Run the free AI crawler check: it reads your live robots.txt and reports whether OAI-SearchBot is allowed or blocked.
- Look for OAI-SearchBot in your server or CDN logs over the following weeks to see whether requests stop.
robots.txt is a public request, not access control. OpenAI publishes its own documentation for OAI-SearchBot; confirm the current token and behaviour there, since operators change them.
- Check which AI bots can read your siteTests your live robots.txt against every major AI agent.
- Block AI crawlersGenerate robots.txt rules for training bots, search bots or both.
- Every AI crawler user agentThe full list with operators and purposes.
- GPTBot vs ChatGPT-UserWhy OpenAI uses more than one agent.
Related explainers
- Why is my page not indexed
- robots.txt vs noindex
- Missing canonical
- Free technical SEO audit
- AI crawler user agents
- GPTBot user agent
- ChatGPT-User user agent
- ClaudeBot user agent
- Claude-SearchBot user agent
- PerplexityBot user agent
- Google-Extended user agent
- Applebot-Extended user agent
- CCBot user agent
- Bytespider user agent
- Meta-ExternalAgent user agent
- Amazonbot user agent
- SEO MCP server
- 404 vs 410
- hreflang vs canonical
- What is a canonical tag