What Claude-SearchBot does
Crawls pages to improve the search results Claude can return and cite.
Allow it if you want Claude to find and cite your pages, even if you block ClaudeBot for training.
How to block Claude-SearchBot in robots.txt
Add these two lines to the robots.txt file at the root of your domain: User-agent: Claude-SearchBot and, on the next line, Disallow: /
To block only part of the site, replace / with a path such as /private/. Rules apply per host, so a subdomain needs its own robots.txt.
How to allow Claude-SearchBot
If you have no rule for Claude-SearchBot and no blanket User-agent: * Disallow: /, it is already allowed. To allow it explicitly while blocking other bots, add User-agent: Claude-SearchBot followed by Allow: /
Check it worked
- Publish the updated robots.txt and open it in a browser at /robots.txt to confirm the live file has the rule.
- Run the free AI crawler check: it reads your live robots.txt and reports whether Claude-SearchBot is allowed or blocked.
- Look for Claude-SearchBot in your server or CDN logs over the following weeks to see whether requests stop.
robots.txt is a public request, not access control. Anthropic publishes its own documentation for Claude-SearchBot; confirm the current token and behaviour there, since operators change them.
- Check which AI bots can read your siteTests your live robots.txt against every major AI agent.
- Block AI crawlersGenerate robots.txt rules for training bots, search bots or both.
- Every AI crawler user agentThe full list with operators and purposes.
- GPTBot vs ChatGPT-UserWhy OpenAI uses more than one agent.
Related explainers
- Why is my page not indexed
- robots.txt vs noindex
- Missing canonical
- Free technical SEO audit
- AI crawler user agents
- GPTBot user agent
- OAI-SearchBot user agent
- ChatGPT-User user agent
- ClaudeBot user agent
- PerplexityBot user agent
- Google-Extended user agent
- Applebot-Extended user agent
- CCBot user agent
- Bytespider user agent
- Meta-ExternalAgent user agent
- Amazonbot user agent
- SEO MCP server
- 404 vs 410
- hreflang vs canonical
- What is a canonical tag