What this check inspects
Shipwork fetches the page and reads its meta robots tags and any X-Robots-Tag header, then resolves robots.txt for the exact path and query using the standard matching rules. It follows up to three same-origin sitemaps, including a couple of children of an index, and checks whether the URL is listed. It also resolves robots.txt for the same-origin stylesheets and scripts the page needs, to catch a block that stops a search engine rendering the page. It reports an explicit index and noindex on the same page as a contradiction, and it reads the exact combination of a robots block, a noindex and a sitemap listing.
What a failure means
A robots.txt block combined with a noindex is an error: the crawler cannot fetch the page, so it never sees the request to drop it, and the URL can stay listed. A robots.txt rule that blocks a stylesheet or script the page needs is an error, because the crawler sees a broken or incomplete page. A page that carries both an index and a noindex directive is an error, since the contradictory requests can be ignored together. A sitemap that lists a blocked URL, and a sitemap that lists a noindexed URL, are warnings. A page that could not be fetched, or one that returns 400 or above, is an error.
How to fix it
- Never combine a robots.txt block with a noindex on the same URL. If the goal is removal from search, allow crawling and keep the noindex so it can be read.
- Remove blocked stylesheets and scripts from the Disallow rules, or the crawler cannot render the page.
- Keep one index directive and remove the other when both appear in meta robots or the X-Robots-Tag header.
- Take blocked or noindexed URLs out of the sitemap, or change the signal so the sitemap and the page agree.
A typical failure, worked through
The setupA team wants a landing page kept out of search, so it both disallows the path in robots.txt and adds a noindex meta tag. The page is still listed in the sitemap from an earlier campaign.
What the check reportsThe check reports the robots block plus the noindex as an error, the sitemap listing as a warning, and a blocked script that the page needs to render as an error.
The pointEach signal alone would do something reasonable. Together they cancel the intended effect, and only checking them against each other reveals it.