GPTBot vs ChatGPT-User: Which to Allow?

GPTBot versus ChatGPT-User: allow one, maybe block the other

Blocking “AI crawlers” as one lump is the costliest robots.txt mistake a store can make right now. The agents do different jobs with opposite business value: one trains a model you will never see, the other answers a shopper holding a wallet. Here is each agent, what it does, and the per-agent policy that serves stores best.

By Hazzy · Software engineer, former SEO specialist

The agent lineup

Agent	Owner	Job
GPTBot	OpenAI	Bulk crawling for model training
ChatGPT-User	OpenAI	Live fetch to answer a user, with citation link
ClaudeBot	Anthropic	Training plus assistant grounding
PerplexityBot	Perplexity	Grounding shopping answers with citations
Google-Extended	Google	Opt-out signal for AI training use
Applebot	Apple	Feeds Apple intelligence features
CCBot	Common Crawl	Open dataset half the industry trains on

Blocking training crawlers is defensible. Blocking search-time agents deletes you from answers that would have sent cited shoppers.

Training crawlers

GPTBot, CCBot and Google-Extended exist to build models, not to send you visitors. Blocking them is a legitimate, defensible choice (many publishers do), and Google-Extended exists precisely as that opt-out. The cost is abstract and future-facing: less of your content in future models. Make the call deliberately, document it, and move on.

Search-time agents

ChatGPT-User and PerplexityBot fetch your pages at answer time to ground a response with a link back to you. Blocking them does not protect anything (your pages are public). It only deletes you from answers that would have delivered qualified shoppers. This is the block that costs real revenue, and it is very often set accidentally by a blanket “block AI” snippet.

The revenue test

Ask of each agent: does allowing it send me shoppers? Training crawlers: no. Search-time agents with citations: yes. Set policy by that answer, not by how you feel about AI training.

A sane per-agent policy

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Allow: /products/
Allow: /collections/

User-agent: PerplexityBot
Allow: /products/
Allow: /collections/

User-agent: ClaudeBot
Allow: /products/
Allow: /collections/

Training crawlers refused, search-time agents welcomed on the money pages. Adapt the paths to your catalogue, keep the principle. And remember the matching rules: a stale narrow block for an agent overrides your shiny new allow. Verify the resolved outcome per agent, not the file at a glance.

Deeper: robots.txt for AI crawlers · llms.txt guide · Related guide: AI crawlers blocked

Find out if this is happening to you

Shipwork resolves your robots.txt per agent with longest-match precedence and tells you exactly which of the nine named AI crawlers are allowed or refused. Free, no account, no signup. Paste your store address.

Check my agents

Questions

What is the difference between GPTBot and ChatGPT-User?

GPTBot bulk-crawls for training; ChatGPT-User fetches a live page to answer a specific user, with a citation link. One costs you nothing to block, the other costs you traffic.

Does PerplexityBot send traffic?

Yes: Perplexity answers cite sources with links. Blocking it removes you from shopping answers that convert.

What does Google-Extended control?

Whether your pages may be used to improve Google AI models. It is an opt-out signal, separate from Search indexing.

Can I allow an agent only on product pages?

Yes: Allow rules take paths, so you can welcome agents on /products/ and /collections/ while refusing the rest. Longest match wins within the agent group.

Keep reading

Robots.txt for AI Crawlers: Allow GPTBot AI Crawler Access on Online Stores: 2026 Data AI Crawler User Agents: Current Bot List and Robots.txt How to Block GPTBot: Robots.txt, Cloudflare and Nginx