AI crawler access on online stores: what 963 stores showed

How many stores block AI crawlers

The 2026 study drew a random sample of stores from the Tranco top one million domains, kept the ones selling on Shopify or WooCommerce, and read twenty pages per store from outside its own network in one crawl in September 2026, the way a crawler sees them. Of the 963 stores audited, 2.9% disallow at least one AI crawler in robots.txt, 28 stores in all. The training crawler behind one major assistant is disallowed by 1.7% of the 963 stores checked, 16 stores. Blocking is the exception rather than the rule: few of the stores audited disallowed any AI crawler at all.

That is an outside reading of the rule file, not a record of refused requests. A store can name an agent in robots.txt and still serve it, and a store that names no agent can still stop one at the network. Read the table as what the file says.

It helps to separate the crawlers by job. Some collect pages to build training data, some fetch a single page to ground a live answer, and some feed an open dataset many systems draw on. The block that matters most to a store is the second kind, because a crawler that cannot read a product page cannot put that product into an answer. The metrics below cover the broad case, the narrow training case, and the file that describes the store to assistants.

Shopify and WooCommerce split

Blocking is not spread evenly. In the sample, 7.6% of 250 WooCommerce stores blocked at least one AI crawler, against 1.3% of 713 Shopify stores. The narrow training block shows the same shape: 4.4% against 0.7%. The counts are small, so read the gap as a lead rather than a verdict, but it is consistent across both block metrics.

The split follows where the file comes from. On WooCommerce, robots.txt is a file that WordPress and its plugins can write, so a blanket rule can arrive with a security or SEO plugin and stay there. On Shopify the file is produced from an editable template, so a block usually has to be added on purpose. When a platform makes a rule easy to inherit, more stores inherit it by accident.

Who publishes an llms.txt file

Publishing an llms.txt file was the common case: most of the 963 audited stores served one at the site root, 79%, or 765 stores. The platform split runs the other way from the block metrics. On Shopify, 99% of 713 stores served the file, almost every store in the group. On WooCommerce, 24% of 250 stores did, about a quarter.

The file is a young convention. Some assistants read it and many do not, and it is not an indexing signal, so treat it as a description of the store rather than a gate. What it changes is how accurately an assistant can describe the store when it does read: a short summary and a list of the pages that matter, in plain text, at one address. It complements robots.txt. One says who may read, the other says what is here.

What a block costs in AI answers

When a store disallows an assistant's crawler, an assistant that honours the file cannot read its pages, so it cannot quote or cite them in a shopping answer. The store does not disappear from ordinary search results: the loss is the answer surface, where a shopper asks for a recommendation and gets a short list of options. A store that cannot be read cannot appear in that list.

The size of the loss depends on the agent. A block on a crawler that only collects training data is a policy choice about reuse, and a defensible one. A block on a crawler that fetches pages to ground a live answer removes the store from the answer itself, which is the block to review first. The two are separate controls even when they come from the same provider, so allowing one does not force the other.

One limit applies to every block. Robots.txt is a request: compliant crawlers honour it, others ignore it, and a user agent string is easy to copy. A robots.txt rule alone does not stop a request reaching the server, and a firewall rule that blocks a request can refuse a crawler the file allows. Permission and enforcement are separate layers, and a clean robots.txt coexists with invisible products more often than owners expect.

How to decide what to allow

1. Read the live robots.txt and list which named agents it disallows. Do not trust the file you remember; templates and plugins rewrite it.
2. Sort the list by job: training crawlers, answer crawlers, open dataset crawlers. Name the job before deciding, because the same name can cover a fetch that cites you and a crawl that does not.
3. Allow the agents that fetch a page to answer a question and link the source. Being readable is the price of being cited.
4. Make a deliberate call on training crawlers, and write it down so a later theme change or copied snippet does not silently reverse it.
5. Re-check after every theme, app or plugin update, and after a platform change, because on many stores the file is generated rather than stored.
6. Check enforcement as well as permission: a CDN or firewall can refuse an agent the file allows.

Run the AI crawler check to see which agents your own file allows. For the mechanics, read robots.txt for AI crawlers on Shopify, and for the describing file read what llms.txt is and how to add it. Both walk through platform specifics: the Shopify template, the WordPress file, and how to verify what is actually served.

What the check measures and misses

The check reads the store's live robots.txt and resolves it for a fixed list of named AI agents against product paths, then reports which are allowed and which are refused. It reads the rules the way a crawler would, including the ordering that decides which group wins. It cannot see a request that never reached robots.txt, a challenge added by a CDN, or a crawler that chooses to ignore the file. It is one reading at one time, so a rules change after the crawl is not reflected.

The sample, the intervals and the platform cuts behind this article are in The State of Ecommerce SEO 2026. The same numbers with their bases are on the statistics page.

Questions

Keep reading