Can Googlebot fetch this URL?
Enter a site and the path to test, and pick the agent. Shipwork fetches the live robots.txt from outside your network and resolves that one URL the way the agent does, naming the exact line that decides it.
How this resolves
It fetches /robots.txt once, then applies the standard precedence: the most specific matching user-agent wins, inside that group the longest matching path wins, and Allow beats Disallow on a tie. The result names the exact line responsible, so a block is never a mystery. If there is no robots.txt, nothing is disallowed and the test says so. To inspect a whole file rather than one URL, use the robots.txt simulator.
Questions
- How is this different from reading robots.txt?
- Reading the file tells you the rules. This answers the question those rules resolve to for one specific URL and one specific agent, which is the thing that actually decides whether a page can be crawled.
- Which agent does it use?
- Googlebot by default, but you can name any agent. Named agents are matched most-specific-first, so a GPTBot group overrides the * group for GPTBot and only for GPTBot.
- What if the site has no robots.txt?
- Then nothing is disallowed. A missing file (HTTP 404) means every URL is allowed, and the test says so rather than guessing.
- Does it store anything?
- No. It fetches robots.txt from outside your network, resolves the path, and returns the verdict. Nothing is saved.