Does robots.txt say what you think it says?

A single "Disallow: /" left over from a staging setup removes an entire site from search, invisibly. And a robots.txt with no Sitemap line makes crawlers work harder to find your pages.

Reads your robots.txt from outside your network and resolves what it means for crawlers. Saved to your account if you are signed in to a paid plan.

How this check works

Shipwork fetches your robots.txt and parses it the way a crawler does, using longest-match rule resolution. It reports a site-wide block, a missing sitemap reference, and anything that would stop crawlers reaching your pages. Robots.txt is a request, not a lock, but it is the first gate.

How to check robots.txt

Check the live file from outside your network, then see how its rules affect crawling. A robots.txt block prevents crawling; it does not reliably remove a URL that is already indexed.

Enter your site’s public domain and run the robots.txt check.
Review whether /robots.txt was found and whether it contains a rule that blocks all crawlers. A missing file is not itself a block.
Check that the file includes a Sitemap: line pointing to the sitemap you publish.
If you need to test a particular path or user agent, use a robots tester or the dedicated AI crawler check; this site-wide check does not simulate every path-agent combination.
After changing the file, run the check again and verify the served text reflects the change.

Errors this check finds, and their fixes

Blocked by robots.txt: When a robots.txt rule blocks real pages, and how to find which rule

Questions

What does a disallow-all look like?

A group with User-agent: * followed by Disallow: /. It tells every compliant crawler to skip the whole site.

Does robots.txt affect ranking?

Indirectly and seriously: blocked pages cannot be crawled, so they cannot be indexed or rank.

Should robots.txt list my sitemap?

Yes. A Sitemap: line lets crawlers find it immediately instead of relying on convention.

Keep reading

What this reads

Reads public pages from outside your network, the way a crawler does.
No signup, and nothing is stored on the free checks.
One of 83 checks across discovery, content, markup, trust and catalogue data.

83checks

0signups needed

0stored on free checks

Cool, just ran an audit and got some insights! Let’s see if I can fix them :)

Tenta Dev, @DevTenta

Whoah! this site is crazy, nothing to roast at all

Ainreek, @AinreekAI

cool as heck, tried it on my blog and the insights it gave are helpful and not bluff.

Refaat 🍉 (Mouse), @1Mouse__