Which AI crawlers are checked?

Nine named agents including GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot and CCBot — each resolved with its own longest-match robots.txt rules.

Should I block AI crawlers?

Blocking training crawlers can be legitimate. Blocking search-time agents that cite sources removes you from shopping answers that would send traffic. Decide per agent.

Is robots.txt the only gate?

No. CDN bot protection can refuse an agent that robots.txt allows. Shipwork also reports whether llms.txt exists.

Can AI search actually read your store?

More people now ask an assistant what to buy than read a results page. If your robots.txt blocks those crawlers — and plenty of stores block them without meaning to — you are invisible in that answer. This reads your robots.txt the way each assistant does.

Reads your public robots.txt and applies each agent's own rules, including longest-match precedence. Nothing is stored.

Questions

Which AI crawlers are checked?: Nine named agents including GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot and CCBot — each resolved with its own longest-match robots.txt rules.
Should I block AI crawlers?: Blocking training crawlers can be legitimate. Blocking search-time agents that cite sources removes you from shopping answers that would send traffic. Decide per agent.
Is robots.txt the only gate?: No. CDN bot protection can refuse an agent that robots.txt allows. Shipwork also reports whether llms.txt exists.