Does robots.txt say what you think it says?
A single "Disallow: /" left over from a staging setup removes an entire site from search, invisibly. And a robots.txt with no Sitemap line makes crawlers work harder to find your pages.
Reads your robots.txt from outside your network and resolves what it means for crawlers. Nothing is stored.
How this check works
Shipwork fetches your robots.txt and parses it the way a crawler does, using longest-match rule resolution. It reports a site-wide block, a missing sitemap reference, and anything that would stop crawlers reaching your pages. Robots.txt is a request, not a lock — but it is the first gate.
Questions
- What does a disallow-all look like?
- A group with User-agent: * followed by Disallow: /. It tells every compliant crawler to skip the whole site.
- Does robots.txt affect ranking?
- Indirectly and seriously: blocked pages cannot be crawled, so they cannot be indexed or rank.
- Should robots.txt list my sitemap?
- Yes. A Sitemap: line lets crawlers find it immediately instead of relying on convention.