| User agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training |
| OAI-SearchBot | OpenAI | AI search |
| ChatGPT-User | OpenAI | User-triggered fetch |
| ClaudeBot | Anthropic | Training |
| Claude-SearchBot | Anthropic | AI search |
| PerplexityBot | Perplexity | AI search |
| Google-Extended | Training control | |
| Applebot-Extended | Apple | Training control |
| CCBot | Common Crawl | Training (open dataset) |
| Bytespider | ByteDance | Training |
| Meta-ExternalAgent | Meta | Training |
| Amazonbot | Amazon | AI assistant |
Training, search and user fetches are different
Training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent) collect content for building models. AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index pages so assistants can cite and link them. User-triggered agents (ChatGPT-User) fetch one page because a person asked. Many sites block training and allow search, so their pages can still be cited.
Control tokens are not crawlers
Google-Extended and Applebot-Extended never request your pages. Googlebot and Applebot crawl as usual, and these tokens only tell the operator whether that content may be used for AI training. Blocking them does not affect Google or Apple search.
Each user agent in detail
- GPTBotOpenAI: training
- OAI-SearchBotOpenAI: ai search
- ChatGPT-UserOpenAI: user-triggered fetch
- ClaudeBotAnthropic: training
- Claude-SearchBotAnthropic: ai search
- PerplexityBotPerplexity: ai search
- Google-ExtendedGoogle: training control
- Applebot-ExtendedApple: training control
- CCBotCommon Crawl: training (open dataset)
- BytespiderByteDance: training
- Meta-ExternalAgentMeta: training
- AmazonbotAmazon: ai assistant
Last reviewed 2026-11-15 against each operator’s public documentation. Tokens and policies change; check the operator’s page before relying on a rule.
- Check which AI bots can read your siteTests your live robots.txt against every major AI agent.
- Block AI crawlersGenerate robots.txt rules for training bots, search bots or both.
- GPTBot vs ChatGPT-UserWhy OpenAI uses more than one agent.
Related explainers
- Why is my page not indexed
- robots.txt vs noindex
- Missing canonical
- Free technical SEO audit
- GPTBot user agent
- OAI-SearchBot user agent
- ChatGPT-User user agent
- ClaudeBot user agent
- Claude-SearchBot user agent
- PerplexityBot user agent
- Google-Extended user agent
- Applebot-Extended user agent
- CCBot user agent
- Bytespider user agent
- Meta-ExternalAgent user agent
- Amazonbot user agent
- SEO MCP server
- 404 vs 410
- hreflang vs canonical
- What is a canonical tag