X-Robots-Tag checker
The X-Robots-Tag is an HTTP response header that gives search engines the same instructions as a meta robots tag, including noindex. Because it is a header, you never see it on the page. This reads it from outside your network.
A set of your key pages plus robots.txt, fetched from outside your network. Saved to your account if you are signed in to a paid plan.
How this check works
Shipwork requests a set of your key pages and reads the X-Robots-Tag header in each response, alongside the meta robots tag. It flags any page whose header tells crawlers not to index it.
Questions
What is the X-Robots-Tag?
How do I see the X-Robots-Tag?
Where does an unwanted X-Robots-Tag come from?
Keep reading
What this reads
- Reads public pages from outside your network, the way a crawler does.
- No signup, and nothing is stored on the free checks.
- One of 83 checks across discovery, content, markup, trust and catalogue data.
Cool, just ran an audit and got some insights! Let’s see if I can fix them :)
Whoah! this site is crazy, nothing to roast at all
cool as heck, tried it on my blog and the insights it gave are helpful and not bluff.