What does your robots.txt actually block?
Paste the file and read it the way a crawler does: per user-agent group, with the real longest-match precedence. Then test any URL against any agent and see the exact line responsible.
How this works
It parses your file into user-agent groups and resolves each URL the way a crawler does: the most specific matching agent wins, and inside that group the longest matching path wins, with Allow beating Disallow on a tie. It then flags the mistakes that quietly remove a site from search: a bare Disallow: /, blocking the CSS and JavaScript Google needs to render, blocking the sitemap, and a missing or stale Sitemap: line. Nothing here leaves the page. Once your file is right, run the single-URL test against the live site.
Questions
- What does robots.txt actually control?
- Which paths crawlers may request. It is a request, not a lock: well-behaved crawlers obey it, malicious ones ignore it. It manages crawling, it does not secure anything.
- Which group applies to a crawler?
- The most specific user-agent that matches wins. A group naming GPTBot beats a group for *, and a prefix such as Googlebot also matches Googlebot-News. If no named group matches, the * group applies.
- Allow and Disallow both match a URL. Which wins?
- The longer, more specific path wins. When two rules match and have the same length, Allow beats Disallow. That tie-break is why a single Allow can rescue one path out of a broad Disallow.
- Does this send my robots.txt anywhere?
- No. It is parsed and matched entirely in your browser. Nothing is uploaded, logged or stored.