How to tell whether AI assistants are allowed to read your product pages

What the error means

Your robots.txt disallows one or more of the crawlers that assistants use, so those assistants cannot read your catalogue and will not cite your products.

Why Google rejects it

Most stores never chose this. A blanket disallow written for scrapers catches GPTBot, ClaudeBot and PerplexityBot as collateral, and the agents differ in what they do: some fetch for training, some fetch to answer a live question with a link back. Blocking the second kind removes you from answers that would have sent traffic.

How to fix it

1. Read your robots.txt and find which agents are disallowed. The rules that match an agent most specifically win, so a broad allow does not override a narrow block.
2. Decide per agent rather than in one sweep. Search-time agents that cite sources are worth allowing even if you refuse training crawlers.
3. On Shopify, robots.txt comes from a template you can edit — the rules are not fixed.
4. Remember robots.txt is not the only gate. Bot protection at the CDN can refuse an agent that robots.txt allows.

Common causes

Search crawler is disallowed

OpenAI's OAI-SearchBot is the crawler used to help surface websites in ChatGPT search. A robots.txt group that disallows this token can prevent its automated crawl, even if other bots such as Googlebot are allowed. Decide whether you want search discovery, training exclusion, or both; these are separate choices.

Training and search tokens were conflated

GPTBot is used for potential model training, while OAI-SearchBot supports search features. Blocking GPTBot does not by itself block OAI-SearchBot, and allowing one does not grant the other permission. A generic “AI bot” rule or copied template can obscure which token is actually affected.

Firewall challenges the crawler

robots.txt expresses a crawl preference but does not open a connection through a CDN, WAF, or hosting firewall. A bot manager can return 403, 429, or a challenge page to the request even when the robots file allows the agent. Compare the live response with provider logs and use verified crawler guidance before creating allow rules.

Product content is not public HTML

An assistant may be unable to read content that requires a customer account, a store password, or a client-side interaction to reveal. A product page that returns a shell without its title and description may also be hard for a crawler to interpret. Check the anonymous page response, not the authenticated admin preview.

Wrong assumption about user-triggered fetches

ChatGPT-User is used for certain user-initiated page visits and is distinct from the automated search crawler. Its access behavior does not decide whether a page appears in ChatGPT search. Confirm which product surface and crawler you mean before changing robots rules.

Fix it on your platform

On Shopify

1. Visit the live robots.txt on the primary domain and inspect the groups for OAI-SearchBot and GPTBot. Shopify serves a default file; if it is customized, the published theme contains the robots.txt.liquid template under Online Store > Themes > Edit code.
2. Make a deliberate per-agent decision. If you want ChatGPT search discovery, do not disallow OAI-SearchBot on the product paths you want found. You can still decide separately whether GPTBot may crawl content for training.
3. Check the store's password mode, product visibility, and Online Store sales channel. An unpublished or password-protected product is not a public page that a crawler can read.
4. If robots.txt allows the crawler but requests fail, review the CDN or bot-protection app rules and logs. Do not rely on a user-agent string alone as identity; use the provider's current verified-crawler guidance where available.

On WooCommerce

1. Open the live /robots.txt and inspect the exact user-agent groups. Review directives added by SEO, security, or robots editor plugins; check both the physical file and any virtual WordPress output.
2. Allow OAI-SearchBot on the public product routes if ChatGPT search visibility is your goal. Set GPTBot independently according to your training preference rather than using one blanket AI-crawler rule.
3. Make sure products are published and accessible without login. Check the public page source for the product name, description, and links instead of testing only while signed in to WordPress.
4. Inspect security plugin and CDN events for blocked crawler requests. If you add an allow rule, use current official crawler identification guidance and avoid trusting spoofable user-agent text by itself.

On other platforms

1. Use the platform's robots.txt or search visibility controls to check OAI-SearchBot and GPTBot separately. If robots customization is unavailable, ask platform support where its generated robots rules are managed.
2. For Wix, BigCommerce, and Squarespace, confirm the storefront is published on its permanent domain, product pages are publicly visible, and no password or member-only restriction covers them.
3. On custom hosting, allow the intended crawler token in robots.txt for the relevant public paths, then verify the server and CDN also serve the page without a challenge.
4. Keep user-triggered access separate from automated search crawling. A page that a user can explicitly ask ChatGPT to visit is not proof that the automated search crawler can discover or index it.

How to confirm it is fixed

1. Fetch the live robots.txt and check the effective rule for OAI-SearchBot on a representative product URL. Review GPTBot separately if you also want to control training use.
2. Open a product page while logged out and confirm its main content is present in the response without a store password, customer account, or browser-only interaction.
3. Check CDN, firewall, and hosting logs for crawler requests and their status codes. Apply current verified-bot guidance from the provider before allowing network traffic.
4. Run the AI crawler check to inspect the robots rules for named assistants, then recheck the live site after any theme or firewall change.
5. Allow time for policy and crawler systems to update. Robots permission allows crawling; it does not guarantee a citation, search appearance, or answer inclusion.

A worked example

Hypothetical case: a Shopify outdoor gear shop blocks GPTBot because the owner does not want product copy used for training. A copied rule also blocks OAI-SearchBot, and a CDN bot setting challenges both. The owner first decides that search visibility is useful but training use is not, then updates the robots template with separate rules and checks the provider's crawler documentation before changing the CDN policy. They test a public product URL and review firewall logs for a successful fetch. The shop has made its pages eligible for that crawler's use; it has not guaranteed that any answer will mention the products.

Other errors like this