Do your sitemap and index signals agree?

Google learns about a page from several signals: the sitemap, the canonical tag, robots meta, the X-Robots-Tag header. When they disagree, Google trusts the page and ignores the sitemap, so the entry is wasted and the wrong URL gets indexed.

Cross-checks the sitemap, canonicals and robots signals on a set of your pages. Saved to your account if you are signed in to a paid plan.

How this check works

Shipwork fetches a set of your pages and compares four signals per URL: is it in the sitemap, is it noindex (meta or header), does its canonical point elsewhere, and what status it returns. It reports pages in the sitemap that are noindex or canonicalised away, and indexable pages that are missing from the sitemap: the two conflicts behind most "why is this page not indexed" cases.

Errors this check finds, and their fixes

Discovered – currently not indexed: Why Google found your pages but has not crawled them yet, and what moves them
Crawled – currently not indexed: Why Google crawled your page and still chose not to index it
Submitted URL marked noindex: How do you fix “Submitted URL marked noindex” in Search Console?
URL is unknown to Google: What does “URL is unknown to Google” mean, and how do you get it discovered?

Questions

Why would a sitemap URL be noindex?

Usually a template default (a tag page, a filtered collection, a thank-you page) that got swept into the sitemap. Google is told both "crawl this" and "do not index this", and it obeys the noindex.

What is a signal conflict?

Any two signals that disagree about a URL: sitemap says index, robots says noindex; or sitemap lists a URL whose canonical points elsewhere. Google resolves the conflict against the sitemap.

Does this need Search Console?

No. It reads the public sitemap, canonicals, robots meta and headers from outside your network, the way a crawler does.

Keep reading

What this reads

Reads public pages from outside your network, the way a crawler does.
No signup, and nothing is stored on the free checks.
One of 83 checks across discovery, content, markup, trust and catalogue data.

83checks

0signups needed

0stored on free checks

Cool, just ran an audit and got some insights! Let’s see if I can fix them :)

Tenta Dev, @DevTenta

Whoah! this site is crazy, nothing to roast at all

Ainreek, @AinreekAI

cool as heck, tried it on my blog and the insights it gave are helpful and not bluff.

Refaat 🍉 (Mouse), @1Mouse__