ShipworkSite inspection
Checks
Every error, explainedGuides →
Pricing Learn Guides

Do your crawl and indexing signals contradict each other?

A robots.txt block stops the fetch, and a noindex only works after a fetch. Put both on one URL and the noindex is never read, so the page can stay in search with no snippet. This check resolves the two signals against each other, and against the sitemap and the files the page needs to render.

Runs the Robots and noindex conflicts check against a bounded sample of your site, from outside your network. Nothing is stored.

What this check inspects

Shipwork fetches the page and reads its meta robots tags and any X-Robots-Tag header, then resolves robots.txt for the exact path and query using the standard matching rules. It follows up to three same-origin sitemaps, including a couple of children of an index, and checks whether the URL is listed. It also resolves robots.txt for the same-origin stylesheets and scripts the page needs, to catch a block that stops a search engine rendering the page. It reports an explicit index and noindex on the same page as a contradiction, and it reads the exact combination of a robots block, a noindex and a sitemap listing.

What a failure means

A robots.txt block combined with a noindex is an error: the crawler cannot fetch the page, so it never sees the request to drop it, and the URL can stay listed. A robots.txt rule that blocks a stylesheet or script the page needs is an error, because the crawler sees a broken or incomplete page. A page that carries both an index and a noindex directive is an error, since the contradictory requests can be ignored together. A sitemap that lists a blocked URL, and a sitemap that lists a noindexed URL, are warnings. A page that could not be fetched, or one that returns 400 or above, is an error.

How to fix it

  1. Never combine a robots.txt block with a noindex on the same URL. If the goal is removal from search, allow crawling and keep the noindex so it can be read.
  2. Remove blocked stylesheets and scripts from the Disallow rules, or the crawler cannot render the page.
  3. Keep one index directive and remove the other when both appear in meta robots or the X-Robots-Tag header.
  4. Take blocked or noindexed URLs out of the sitemap, or change the signal so the sitemap and the page agree.

A typical failure, worked through

The setupA team wants a landing page kept out of search, so it both disallows the path in robots.txt and adds a noindex meta tag. The page is still listed in the sitemap from an earlier campaign.

What the check reportsThe check reports the robots block plus the noindex as an error, the sitemap listing as a warning, and a blocked script that the page needs to render as an error.

The pointEach signal alone would do something reasonable. Together they cancel the intended effect, and only checking them against each other reveals it.

Questions

Why is Disallow plus noindex worse than either alone?
Disallow stops the fetch, so the crawler never reads the noindex. The URL can stay in search from links alone, listed without a snippet, which is the opposite of what the combination was meant to do.
Why would robots.txt block a stylesheet?
Usually an over-broad rule, for example Disallow: /assets or Disallow: /wp-content, that was meant for something else but also matches the files the page needs to render.
Does the check follow every sitemap?
It follows up to three same-origin sitemaps, including a couple of children when the sitemap is an index, so the check stays fast.

Related checks and guides