What this check inspects
Shipwork fetches /robots.txt from outside your network, parses it into user-agent groups with their Allow, Disallow and Crawl-delay rules, and lists the Sitemap lines. When you name an agent and a path it resolves the outcome the way the standard matching works: the most specific user-agent wins, and inside a group the longest matching path wins, with * matching any run of characters and a trailing $ anchoring the end.
What a failure means
A non-200 response is not a block: crawlers then treat the whole site as allowed, which is the opposite of what most people assume. A file with no parsable rules is a warning. The serious case is a rule that quietly matches more than intended: Disallow: / inside a group blocks the entire site, and because robots.txt stops crawling rather than indexing, pages can drop out of the crawl while staying listed without a snippet.
How to fix it
- Read the rules as groups. A Disallow applies only to the user-agent lines directly above it, up to the next user-agent line.
- A broader rule beats a narrower one only when it is a longer match. Disallow: /product beats Disallow: / for a product URL, so an allow cannot rescue a page from a site-wide block.
- Test a specific path against a specific agent instead of assuming the rule fires the way you meant.
- Keep a Sitemap: line so crawlers do not have to discover URLs by links alone.
A typical failure, worked through
The setupA store wants its internal search kept out of the index and adds Disallow: /search near the top of robots.txt. Later a developer adds Disallow: / as the first rule in the same group while testing and forgets it.
What the check reportsThe check reports a non-empty rule set and, for the path /, the longest match is Disallow: /. Every product page resolves as blocked for all crawlers, though /search was the only thing you meant to block.
The pointThe block is silent in a browser, because robots.txt never stops a human. Only a check that resolves the rules from outside can see it.