The nine agents that matter
Not all AI crawlers do the same job, and the distinction decides your policy. GPTBot crawls for training data; ChatGPT-User fetches live pages to answer a question with a link back. ClaudeBot similarly serves Anthropic, while PerplexityBot exists almost entirely to ground shopping-style answers with citations. Google-Extended controls whether your pages train Google AI models, Applebot feeds Apple intelligence features, and CCBot (Common Crawl) supplies the open dataset half the industry trains on. Blocking a training crawler is a defensible choice. Blocking a search-time agent that cites sources deletes you from answers that would have sent traffic.
How robots.txt matching actually works
Three rules govern every decision, and nearly every misconfiguration violates one of them. First, the crawler uses the most specific group that names it — a block written for GPTBot beats a general allow for everyone. Second, within that group, the longest matching rule wins, so a long Disallow for a path overrides a short Allow for the root. Third, on Shopify the file comes from a template you can edit — the rules are not fixed, and apps or copied snippets can rewrite them without you noticing. A broad “allow everything” line never rescues an agent that has its own narrower block.
The accidental block
Most blocked stores never chose it. The usual suspects: a copied “block all AI scrapers” snippet pasted during a scare, a security app toggle that nobody revisited, or a theme migration that carried old rules forward. The signature is collateral damage — GPTBot, ClaudeBot and PerplexityBot all refused at once, by rules the owner does not remember writing. If your store blocks every AI agent uniformly, assume accident until proven otherwise.
Decide per agent, not in one sweep
- Read your live robots.txt and list which named agents are disallowed.
- Separate training crawlers (GPTBot, CCBot, Google-Extended) from search-time agents (ChatGPT-User, PerplexityBot, ClaudeBot).
- Allow the search-time agents that cite sources — those links are free qualified traffic.
- Make a deliberate call on training crawlers based on your own view of training use, and document it so the next person does not “fix” it back.
- Re-check after every theme or app change that touches robots.txt.
Beyond robots.txt
Robots.txt is a request, not a lock — and it is not the only gate. CDN bot protection can refuse an agent the file allows, which is why a clean robots.txt sometimes coexists with invisible products. Separately, llms.txt is emerging as the way to tell assistants what your site is and which pages matter, though it is not yet widely honoured. Treat robots.txt as permission, bot settings as enforcement, and llms.txt as introduction.
Shipwork reads your robots.txt and resolves the rules for nine named AI agents exactly the way each one does, then tells you which are allowed and which are refused. Free, no account, no signup — paste your store address.
Check my storeQuestions
- How do I allow GPTBot in robots.txt on Shopify?
- Add a group naming GPTBot with an Allow for the paths you want readable, remembering that the most specific group wins — a stale GPTBot block elsewhere overrides a general allow. Then verify with the free check.
- Should I block AI crawlers from my store?
- Blocking training crawlers can be legitimate. Blocking search-time agents that cite sources removes you from shopping answers that would send traffic. Decide per agent.
- Why does ChatGPT say it cannot read my site when robots.txt looks fine?
- Because robots.txt is not the only gate — CDN bot protection can refuse an agent the file allows. Check both layers.
- What is llms.txt and do I need one?
- An emerging convention for describing your site to assistants. Not required and not yet widely honoured, but cheap to add. This site serves one at /llms.txt.
Keep reading