Can each AI assistant read your site?
One verdict per assistant, the exact robots.txt line behind it, and the line to add if you want it to change. Plenty of sites block AI search by accident; this shows which agents can read you and why.
Reads your public robots.txt and resolves it per agent, testing the home page and a product path. Nothing is stored.
How this report works
Shipwork fetches /robots.txt once and parses it the way crawlers do: the most specific group that names an agent wins, and inside that group the longest matching rule wins. Each named agent is resolved against the home page and a product path, which is why one agent can come back "partial": allowed in one place, blocked in another. llms.txt is fetched separately and judged on whether it has a heading and at least one real link. This is a young area, and the rules and agent names change; the verdict reflects the robots.txt served at the moment you run it.
Questions
- What does this report actually decide?
- For each named assistant it resolves your robots.txt the way that crawler does: the most specific group that names it wins, and inside that group the longest matching rule wins. It tests the home page and a product path, then says whether the agent is allowed, blocked, or allowed on one and blocked on the other.
- What does "partial" mean?
- The rules disagree across the paths that matter: the agent can read one part of the site but not another. That is usually an accident, and the fix names the path that is blocked.
- Does allowing AI crawlers hurt my SEO?
- No. AI crawlers and Googlebot are separate user-agents. Blocking a training crawler does not touch rankings, and allowing a search-time agent does not either. Decide per agent.
- Is this the same as the AI crawler check?
- It reads the same robots.txt with the same parser. This report adds a verdict and a concrete fix per assistant, and shows llms.txt in the same view. The AI crawler check is the older, shorter list.