Can each AI assistant read your site?
One verdict per assistant, the exact robots.txt line behind it, and the line to add if you want it to change. Plenty of sites block AI search by accident; this shows which agents can read you and why.
Reads your public robots.txt and resolves it per agent, testing the home page and a product path. Nothing is stored.
How this report works
Shipwork fetches /robots.txt once and parses it the way crawlers do: the most specific group that names an agent wins, and inside that group the longest matching rule wins. Each named agent is resolved against the home page and a product path, which is why one agent can come back "partial": allowed in one place, blocked in another. llms.txt is fetched separately and judged on whether it has a heading and at least one real link. This is a young area, and the rules and agent names change; the verdict reflects the robots.txt served at the moment you run it.
What the AI crawler access checker resolves
The report fetches your robots.txt once, parses it into user-agent groups, and resolves each named agent against two paths: the home page and a product path. It covers agents such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Amazonbot and others, each labelled with the assistant it serves, so it reads as a list of products rather than crawler names. That makes the output readable without knowing crawler filenames.
The decision follows the standard precedence: the most specific group that names the agent wins, and inside that group the longest matching rule wins, with Allow beating Disallow on a tie. Each verdict names the exact rule, or says that no rule matched. It also checks whether an llms.txt exists and whether it has a heading and a real link.
How to read the verdicts
Allowed means no rule on either path blocks the agent. Blocked means both tested paths are blocked. Partial means the rules disagree across the two paths, so the agent can read one part of the site but not another; that split is usually accidental. The summary counts how many agents fall into each state. The two tested paths are representative rather than exhaustive.
A verdict describes the robots.txt served at the moment you ran it. It does not prove any assistant will read, describe or cite your site, because being allowed is only permission to crawl. The agent list changes over time, and an agent not listed may still read you under a wildcard rule that covers several agents at once. A wildcard change can therefore affect more than one agent.
Why AI access matters and how it breaks
A robots file copied from a template, or a plugin that blocks bots, can block an AI search agent by accident while leaving Googlebot untouched. Sites often intend to block a training crawler but also block a search-time agent that would otherwise have cited them. The block is often invisible in the interface of the platform, because it lives in the robots file rather than in a setting.
Agents are separate. Blocking one does not affect another, and blocking a training crawler does not change your search rankings. That is why the report tests each agent on its own rather than giving a single verdict for the whole site, which would hide the accidental blocks. A change made for one assistant should be checked against the others before publication.
Change access deliberately
If an agent you want cannot read the site, add a named group with an Allow for the blocked path, which overrides a broader wildcard disallow. If you intend to block an agent, add a named group with Disallow and keep it specific, so you do not affect other crawlers that share the wildcard group. Because the longest match wins, a short Allow cannot beat a longer, more specific Disallow inside it.
If llms.txt is missing, you can add one that states the site's name and links its key pages; it is optional and not yet read everywhere. After any change to the rules or the file, re-run the report and confirm the verdicts you wanted. The report is cheap to run again, so confirm the outcome rather than assuming the edit took effect.
When to use related checks
Use AI crawler check for the simpler list of verdicts, and llms.txt check to inspect that file on its own. robots simulator reads a whole robots.txt when the same file governs many agents, while robots test confirms one path for one agent against the live rules. The simulator is the better choice when you want to read all of the file at once.
For background, see the AI crawlers and robots.txt guide and the difference between GPTBot and ChatGPT-User. Both explain that these agents are separate from Googlebot and can be allowed or blocked on their own. The fix for ChatGPT not reading a store starts from the same rules and walks through the edit, naming the exact robots.txt line to add.