The main AI crawler tokens
Use the value in the User-agent: line, not an assumed browser string. User-agent strings in server logs often include a browser-like prefix and can change; several provider controls are robots.txt tokens rather than distinct HTTP user agents.
| Token | Operator and role | Practical distinction |
|---|---|---|
GPTBot | OpenAI; automated crawling for content that may be used to train foundation models. | A GPTBot block is a training-use preference. OpenAI documents it separately from search. |
OAI-SearchBot | OpenAI; discovers and surfaces pages in ChatGPT Search. | Blocking it opts pages out of ChatGPT Search results. It is an independent control from GPTBot. |
ChatGPT-User | OpenAI; fetches a page in response to a user action in ChatGPT or a custom GPT. | Not an automatic search crawler. OpenAI says robots.txt may not apply to these user-triggered requests. |
ClaudeBot | Anthropic; gathers content that could contribute to model training. | Separate from Anthropic’s search and user-directed agents. |
Claude-SearchBot | Anthropic; helps improve relevance and accuracy of Claude search responses. | Blocking affects search visibility; it is not the training token. |
Claude-User | Anthropic; retrieves pages in response to user-directed requests. | Separate control for on-demand retrieval. |
Google-Extended | Google; a robots.txt control token for certain Gemini model training and grounding uses. | It is not a separate HTTP crawler user-agent string. Google says it does not affect Google Search inclusion or ranking. |
Applebot / Applebot-Extended | Applebot supports Apple search experiences; Applebot-Extended is the robots.txt control for use of crawled content to train generative foundation models. | Apple documents these as separate controls; allowing the former does not require allowing the latter. |
PerplexityBot | Perplexity; its published crawler documentation identifies it as a crawler used to discover and index web content for search. | Perplexity also documents a separate user-triggered fetch agent. Check its current bot page for the exact current names and policy. |
CCBot | Common Crawl; collects web crawl data for its open corpus, which is used by researchers and other downstream users. | Blocking CCBot controls Common Crawl’s own collection, not every dataset or model that may already have used prior crawls. |
These roles are based on provider descriptions, not guarantees about every downstream use. See the official references: OpenAI crawler overview, Anthropic crawler controls, Google-Extended, Applebot, and Common Crawl’s CCBot page. Perplexity publishes its current user-agent guidance at Perplexity bots.
Copyable robots.txt examples
To block a training token while permitting the provider’s documented search crawler, give each token its own group. Replace the example paths and decisions with your policy:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /
To make an explicit allow for all paths, use Allow: / in that named group. To disallow a provider from the whole site, use Disallow: /. For page-level exclusions, use a path rule such as Disallow: /private/. Rules apply to the host and protocol serving that robots.txt file; check every subdomain separately.
Google-Extended is a token rather than a distinct request user agent, so its control looks like this:
User-agent: Google-Extended Disallow: /
Do not copy all example groups blindly. Decide independently for each provider and purpose. A broad User-agent: * group is not a universal override for a more specific named group. The robots.txt for AI crawlers guide explains group matching and Shopify-specific editing.
What a rule does and does not control
Robots.txt is a public crawl instruction, not access control. It does not protect private data, authenticate a request, or force every bot to comply. Use authentication or server/CDN controls for confidential material. Conversely, a clean robots.txt does not prove a bot can fetch a page: a firewall, challenge, rate limit, or origin error can still prevent access.
Also distinguish crawling from use after a crawl. A robots.txt block is a provider-specific signal about access or use described by that provider; it cannot recall material already collected, control other companies’ datasets, or guarantee how an answer will be generated. OpenAI, for example, documents the search and training controls separately, while user-triggered fetches have their own caveat.
Keep the list current
- Check the provider’s own bot page for its current token, purpose, and policy; avoid trusting a copied third-party list.
- Inspect the live robots.txt for each hostname, then inspect CDN and server rules if provider requests still fail.
- Revisit the policy when a provider adds a new token or changes product behavior. Do not infer a bot’s purpose from its name alone.
- Keep an owner and review date with the policy internally. User-agent inventories are a moving reference, not a one-time setup.
For the store-specific rules and decision points, read robots.txt for AI crawlers on Shopify and GPTBot vs ChatGPT-User.
Shipwork fetches your public robots.txt and evaluates the named AI crawler groups it supports. Free, no account, no signup. Paste your store address.
Check my crawler rulesQuestions
What is the ChatGPT crawler user agent?
Does Google-Extended appear in my server logs?
Does blocking GPTBot block ChatGPT Search?
Does robots.txt stop all AI use of my pages?
Keep reading