Can AI search actually read your store?

More people now ask an assistant what to buy than read a results page. If your robots.txt blocks those crawlers (and plenty of stores block them without meaning to), you are invisible in that answer. This reads your robots.txt the way each assistant does.

Reads your public robots.txt and applies each agent's own rules, including longest-match precedence. Nothing is stored.

Which AI crawler access signals this check reads

The check fetches the public robots.txt and resolves rules for nine named agents against the product path /products/. It reports whether each agent is allowed or blocked, which matching rule produced that verdict, and whether the rule came from an agent-specific group or the wildcard group. The matching logic uses the most specific user-agent group and the longest matching path rule; when two matching paths have equal length, Allow wins.

It also checks whether /llms.txt exists. When present, the result reports whether it is empty and how many markdown page links it contains. llms.txt is optional and is not honored by every assistant. Finding it does not mean an AI service will cite or recommend the store.

GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot and Bytespider.
The robots.txt decision for the tested product path, including the matching rule text.
The HTTP result for robots.txt and whether llms.txt is present, empty or contains page links.

How to interpret allowed and blocked

Allowed means the robots.txt rules Shipwork fetched do not disallow that agent for the tested path. Blocked means a matching Disallow rule applies. Read the displayed rule before changing anything: a specific group can override a wildcard group, and a longer matching path can take precedence over a shorter rule. A block may be intentional, especially for training crawlers; decide separately whether you want search-time retrieval agents to access product pages.

This check evaluates robots.txt, not every layer between a crawler and your site. A WAF, CDN, server error, authentication wall or JavaScript-only content can still prevent an agent from reading a page that robots.txt allows. The report also tests one path rather than every URL on your domain. Use it to spot robots rules that need review, then verify the actual pages and security configuration.

An absent robots.txt does not create a disallow rule in this checker; no matching rule resolves as allowed.
A blocked verdict names the Disallow pattern behind it when one matched.
A found llms.txt is descriptive evidence only, not a ranking or citation signal guarantee.

Change robots.txt with a narrow rule

Before editing, decide which agents you want to allow and whether the rule should apply to all public pages or only a section. Preserve blocks you intentionally use. Avoid adding a broad Disallow: / as a quick fix: for the relevant group, it blocks the entire site path space. Robots rules are requests to crawlers, not access controls; private pages still need authentication.

Robots files are plain text at the site root. For example, this group permits OAI-SearchBot to crawl product paths. Check the complete file for duplicate or conflicting groups, publish it, then rerun the checker to confirm the live response and verdict.

User-agent: OAI-SearchBot
Allow: /products/
User-agent: *
Disallow: /private/
Use the robots path tester when you need a verdict for a specific agent and URL path.

When to run an AI crawler check

Run it after changing robots.txt, enabling a bot protection setting, migrating platforms, or seeing that an assistant cannot retrieve public product information. Shopify and other hosted platforms may generate robots.txt from platform settings or theme code, so make changes at the source your platform supports rather than relying on an edit that will be overwritten. The interface and available controls differ by platform; if you are unsure, check the platform’s current documentation.

A robots allow is only the first access check. A public catalogue also needs accurate product data, reachable pages and content a crawler can read. Use the related tools to investigate those separate questions.

AI access report: adds assistant-specific verdict context and llms.txt detail.
Feed check: checks the visible product catalogue data.
Product page check: tests outside reachability of product URLs.

Errors this check finds, and their fixes

AI crawlers blocked: How to tell whether AI assistants are allowed to read your product pages

Questions

Which AI crawlers are checked?

Nine named agents including GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot and CCBot: each resolved with its own longest-match robots.txt rules.

Should I block AI crawlers?

Blocking training crawlers can be legitimate. Blocking search-time agents that cite sources removes you from shopping answers that would send traffic. Decide per agent.

Is robots.txt the only gate?

No. CDN bot protection can refuse an agent that robots.txt allows. Shipwork also reports whether llms.txt exists.

How do I check whether ChatGPT can crawl my website?

Inspect robots.txt for rules applying to the relevant OpenAI crawler and confirm that public pages can be fetched. Shipwork resolves nine named agents against a product path and shows the matching robots rule. The result does not test every URL, WAF response or whether ChatGPT will cite the site. If access matters, also check that the destination page responds and contains readable product information.

Does blocking GPTBot also block ChatGPT search?

Not necessarily. OpenAI uses distinct crawler identities for different purposes, including training and search. A robots rule can name one agent without naming another, so inspect the individual verdicts rather than assuming one block controls all OpenAI access. The crawler names and product behavior can change; consult current OpenAI crawler documentation before making a policy decision.

What does llms.txt do, and do AI crawlers require it?

llms.txt is an optional text file intended to describe a site and point assistants to useful pages. This check reports whether the file exists and whether it contains links, but presence is not proof that an assistant reads or follows it. Robots.txt is the access signal this check resolves; llms.txt should not be treated as a substitute for accurate pages or crawler access.

Why does an AI crawler check say allowed when the assistant cannot read my page?

This check evaluates robots.txt for one product path. It does not test every page or all delivery layers. A WAF, CDN challenge, rate limit, authentication requirement, server error or content rendered only by scripts can still interfere. Check the exact page response and security logs, then test the specific path with the robots path tester if the rule match is unclear.

Should I block AI crawlers from my online store?

That depends on whether you want each service to use your public pages for its stated purpose. Some owners allow search-time retrieval while disallowing training crawlers; others prefer a broader block. Review agent-specific rules and the service’s current documentation, then make a deliberate choice. A robots block is a crawler request, not a way to secure private or paid content.