Can Googlebot fetch this URL?

Enter a site and the path to test, and pick the agent. Shipwork fetches the live robots.txt from outside your network and resolves that one URL the way the agent does, naming the exact line that decides it.

How this resolves

It fetches /robots.txt once, then applies the standard precedence: the most specific matching user-agent wins, inside that group the longest matching path wins, and Allow beats Disallow on a tie. The result names the exact line responsible, so a block is never a mystery. If there is no robots.txt, nothing is disallowed and the test says so. To inspect a whole file rather than one URL, use the robots.txt simulator.

What the robots.txt URL tester does

Give the tester a site, a path and an agent. It fetches that site's robots.txt once and resolves the single path the way the agent would: the most specific matching user-agent group wins, then the longest matching rule, with Allow beating a Disallow of the same length. It returns whether the URL is allowed and names the line that decided it.

If the site has no robots.txt, nothing is disallowed and the test says so. A not-found response, or another client error other than a rate-limit response, is treated as no restrictions. A server error or a rate-limit response is treated as a temporary full disallow, which is how a crawler reads it. The status of the request is shown so an odd verdict can be traced.

How to read the verdict

The result shows the verdict, the status of the robots.txt request, and the matching line if there is one. When a rule decides the answer, the page shows the file with that line highlighted, so a block is not a mystery. The highlighted line shows that precedence decided the answer, not the order the rules appear in the file. That distinction matters when a file lists a Disallow before the Allow that actually wins.

The verdict is for one path and one agent at the moment you ran it. It does not prove Googlebot will crawl the page, because it does not check noindex, canonical or other signals, and it does not speak for every product that uses a crawler. It answers the robots.txt question specifically. For a whole-file view, the simulator shows every group at once.

Why single-URL results matter

A robots file can allow the whole site yet still block one directory, and a wildcard rule can catch more than it first appears to. Reading the file tells you the rules; resolving one URL tells you what those rules mean for the page you actually care about, which is often the question behind a page that will not appear in search.

A block on a product, category or search path can remove pages from crawling without any visible change to the page itself. The test names the responsible rule, so you can decide whether the exclusion was intended or accidental before you change anything. The same URL can be allowed for one agent and blocked for another, so the test always names the agent it used.

Fix a blocked URL

If the block is wrong, add a more specific Allow above or beside the broad disallow, in the group for that agent, so the longest match now allows the path. Because Allow wins a tie, a path can be freed without removing the broad disallow around it. If the block is intended, keep it and remove any link that assumes the page is crawlable.

Edit the source that generates robots.txt rather than the served file, so a deploy does not overwrite your change. Then run the test again on the same URL and agent to confirm the verdict. For a whole-file review, use the robots simulator and read every group at once. For a single URL, re-run this test after the change instead of trusting the edit.

When to use related checks

Use robots simulator to review every group at once and robots check for the live file. The AI access report resolves the named assistants against your rules, so you can see which ones the rules permit. The simulator explains each group in turn, which helps when a wildcard catches more than intended. Reading the whole file first makes the single-URL result easier to act on.

For the fix workflow, see blocked by robots.txt. The Shopify robots.txt guide and the indexability check cover the surrounding signals that decide whether a page can be indexed beyond the crawl question. The guide explains the rules on that platform and the indexability check shows the other signals that decide indexing, so the crawl result is only one part of the picture.

Errors this check finds, and their fixes

Indexed, though blocked by robots.txt: How a page Google may not crawl still ends up in search, and how to decide what to do

Questions

How is this different from reading robots.txt?

Reading the file tells you the rules. This answers the question those rules resolve to for one specific URL and one specific agent, which is the thing that actually decides whether a page can be crawled.

Which agent does it use?

Googlebot by default, but you can name any agent. Named agents are matched most-specific-first, so a GPTBot group overrides the * group for GPTBot and only for GPTBot.

What if the site has no robots.txt?

Then nothing is disallowed. A missing file (HTTP 404) means every URL is allowed, and the test says so rather than guessing.

Does it store anything?

No. It fetches robots.txt from outside your network, resolves the path, and returns the verdict. Nothing is saved.

How do I test if Googlebot can fetch a URL?

Enter the site, the path and Googlebot. The tester fetches the live robots.txt and resolves that URL, then says whether it is allowed and names the line that decided it. If there is no robots.txt, the URL is allowed. The result names the deciding line so a block can be traced.

What happens if the site has no robots.txt?

Nothing is disallowed. A missing file returns a not-found response, which crawlers read as permission to fetch. The tester reports this rather than guessing at rules that do not exist, and it says the file was not found. A missing file is a normal state, not an error.

Does it test the whole file?

No, it tests one URL for one agent. To see every group and rule in a file, use the robots simulator. To check the same URL against several agents, run the test again with a different agent selected. Each run re-fetches the file, so a recent edit is reflected.

Does it store my data?

No. It fetches robots.txt, resolves the path and returns the verdict. Nothing is saved, and it reads the file from outside your network the way a crawler would, so the result reflects what a crawler is served. It does not keep a history of runs.