The agent lineup
| Agent | Owner | Job |
|---|---|---|
| GPTBot | OpenAI | Bulk crawling for model training |
| ChatGPT-User | OpenAI | Live fetch to answer a user, with citation link |
| ClaudeBot | Anthropic | Training plus assistant grounding |
| PerplexityBot | Perplexity | Grounding shopping answers with citations |
| Google-Extended | Opt-out signal for AI training use | |
| Applebot | Apple | Feeds Apple intelligence features |
| CCBot | Common Crawl | Open dataset half the industry trains on |
Training crawlers
GPTBot, CCBot and Google-Extended exist to build models, not to send you visitors. Blocking them is a legitimate, defensible choice — many publishers do — and Google-Extended exists precisely as that opt-out. The cost is abstract and future-facing: less of your content in future models. Make the call deliberately, document it, and move on.
Search-time agents
ChatGPT-User and PerplexityBot fetch your pages at answer time to ground a response with a link back to you. Blocking them does not protect anything — your pages are public — it only deletes you from answers that would have delivered qualified shoppers. This is the block that costs real revenue, and it is very often set accidentally by a blanket “block AI” snippet.
Ask of each agent: does allowing it send me shoppers? Training crawlers: no. Search-time agents with citations: yes. Set policy by that answer, not by how you feel about AI training.
A sane per-agent policy
User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: ChatGPT-User Allow: /products/ Allow: /collections/ User-agent: PerplexityBot Allow: /products/ Allow: /collections/ User-agent: ClaudeBot Allow: /products/ Allow: /collections/
Training crawlers refused, search-time agents welcomed on the money pages. Adapt the paths to your catalogue, keep the principle. And remember the matching rules: a stale narrow block for an agent overrides your shiny new allow — verify the resolved outcome per agent, not the file at a glance.
Deeper: robots.txt for AI crawlers · llms.txt guide · Related guide: AI crawlers blocked
Shipwork resolves your robots.txt per agent with longest-match precedence and tells you exactly which of the nine named AI crawlers are allowed or refused. Free, no account, no signup — paste your store address.
Check my agentsQuestions
- What is the difference between GPTBot and ChatGPT-User?
- GPTBot bulk-crawls for training; ChatGPT-User fetches a live page to answer a specific user, with a citation link. One costs you nothing to block, the other costs you traffic.
- Does PerplexityBot send traffic?
- Yes — Perplexity answers cite sources with links. Blocking it removes you from shopping answers that convert.
- What does Google-Extended control?
- Whether your pages may be used to improve Google AI models. It is an opt-out signal, separate from Search indexing.
- Can I allow an agent only on product pages?
- Yes — Allow rules take paths, so you can welcome agents on /products/ and /collections/ while refusing the rest. Longest match wins within the agent group.
Keep reading