What this check inspects
Shipwork reads robots.txt through the same parser the other checks use and resolves it for each of the major AI agents: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and Bingbot, plus the wildcard rule that catches agents with no group of their own. It then checks llms.txt through the existing llms check, so a placeholder file is told apart from a useful one. Finally it reads the page itself: an H1, a meta description, a main, article or nav landmark, and at least a couple of hundred characters of text without running JavaScript. It also probes one made-up URL to see whether unknown paths return a real 404 or an app shell with status 200.
What a failure means
A blocked AI crawler is a warning: that agent cannot read the site, and if you want to appear in its answers you have to allow it. A wildcard rule that blocks every unnamed crawler is a warning, so an accidental site-wide block is not mistaken for a deliberate policy. A page with no H1, no meta description, no landmark or almost no raw text is a warning, because answer engines have little to quote. A made-up URL that returns 200 is a warning about a soft 404 or an app shell served for every route. Only a page that could not load at all is an error.
How to fix it
- Allow the AI agents you want to appear in, and leave the others blocked if that is deliberate. Target each agent by name rather than blocking the wildcard.
- Publish an llms.txt at the site root with a level-one heading, a summary line and links to the pages that matter.
- Server-render the heading and the main copy so the page has an H1, a meta description and a landmark before scripts run.
- Return a real 404 for unknown paths rather than a styled fallback page with status 200.
A typical failure, worked through
The setupA store pasted an aggressive robots.txt block while hiding from scraper bots, and it disallows the wildcard for everything except a short allow list. The product copy is also injected after load by a JavaScript framework.
What the check reportsThe check reports that several named AI agents are blocked by the wildcard rule, that llms.txt is missing, and that a plain fetch sees very little text. A made-up URL returns the app shell with 200.
The pointEvery one of these is invisible in a browser. The page loads, the assistant does not. The check reads the site the way an answer engine does.