What this check inspects
Shipwork fetches /llms.txt and checks that it exists, is not empty, opens with a level-one heading naming the site, contains a summary line, and lists links. It also reads robots.txt and reports how many of twelve known AI crawlers are disallowed from the whole site, because an llms.txt has limited effect if those agents cannot read it.
What a failure means
A missing llms.txt is informational, since the convention is optional and not universally honoured. A file that exists but is empty, has no heading, or lists no links is a warning. Blocked AI crawlers are informational but they undercut the file.
How to fix it
- Put the file at the site root as /llms.txt over https.
- Open with a level-one heading naming the site, add a summary line, then list key pages as markdown links.
- Point at pages that exist and return 200. A link to a redirect or a 404 makes the map worse than none.
- Review which AI crawlers robots.txt blocks, and allow the ones that cite sources if you want the traffic.
A typical failure, worked through
The setupA store adds /llms.txt by pasting its footer sitemap, which lists no links and has no heading.
What the check reportsThe check reports the file present but empty of useful structure: no links and no leading heading, so it gives an assistant nothing to follow.
The pointPresence is not the same as usefulness. The links are the whole value of the file.