Is Google actually allowed to index your pages?
A single noindex tag, a stray robots header or a page-level block can remove a page from search without changing how it looks to you at all. This reads your key pages the way a crawler does and reports what it is told.
A set of your key pages plus robots.txt, fetched from outside your network. Saved to your account if you are signed in to a paid plan.
How this check works
Shipwork crawls a set of your key pages and reads the raw HTML and response headers: the same signals a crawler sees before it runs any JavaScript. It reports pages marked noindex, pages returning errors to an outside request, and robots.txt rules that block crawlers from the whole site.
How to check noindex on a website
A noindex directive can appear in a page’s robots meta tag or in its X-Robots-Tag response header. This indexability check reads those signals alongside robots.txt, canonical, sitemap membership, and the page response so you can see the surrounding context.
- Enter the public URL you want to investigate and run the indexability check.
- Look for a noindex finding on the fetched page. Check both its HTML meta robots directive and the response’s X-Robots-Tag header; either can request exclusion.
- Review related findings: robots.txt may stop crawling before a crawler can see noindex, while a canonical to another URL signals a different preferred page.
- If the page should appear in search, remove the unintended noindex at its source and allow crawling. If exclusion is deliberate, keep the directive and avoid listing the URL as an indexable sitemap entry.
- Run the check again to confirm the public response changed. Use Search Console URL Inspection to check Google’s current indexing state; this tool does not query Google’s index.
Errors this check finds, and their fixes
- Excluded by noindex: Which of your pages carry a noindex instruction, and how to remove it
Questions
Why are my pages missing from Google?
Does this work on any platform?
What is the difference between noindex and a robots.txt block?
Keep reading
What this reads
- Reads public pages from outside your network, the way a crawler does.
- No signup, and nothing is stored on the free checks.
- One of 83 checks across discovery, content, markup, trust and catalogue data.
Cool, just ran an audit and got some insights! Let’s see if I can fix them :)
Whoah! this site is crazy, nothing to roast at all
cool as heck, tried it on my blog and the insights it gave are helpful and not bluff.