What does your robots.txt actually block?
Paste the file and read it the way a crawler does: per user-agent group, with the real longest-match precedence. Then test any URL against any agent and see the exact line responsible.
How this works
It parses your file into user-agent groups and resolves each URL the way a crawler does: the most specific matching agent wins, and inside that group the longest matching path wins, with Allow beating Disallow on a tie. It then flags the mistakes that quietly remove a site from search: a bare Disallow: /, blocking the CSS and JavaScript Google needs to render, blocking the sitemap, and a missing or stale Sitemap: line. Nothing here leaves the page. Once your file is right, run the single-URL test against the live site.
What the robots.txt simulator parses
Paste a robots.txt and the tool splits it into user-agent groups, each with its own rules. Consecutive user-agent lines at the top of a group are treated as aliases for the same rule set. It records Allow, Disallow and Crawl-delay lines with their line numbers, and any Sitemap declarations, then shows each group and what it blocks. Groups are read in order until a new user-agent line starts another.
If you also give a URL, it resolves that URL for a chosen agent the way a crawler does: the most specific matching agent group wins, and inside that group the longest matching path wins, with Allow beating Disallow on a tie. The result names the line that decided it and the agent group that matched, so you can see which part of the file applied.
How to read a group and a verdict
Each group lists its rules and counts how many are disallows. A group with no rules permits everything for those agents, which is rarely what an unfinished edit intended, and a file with only Allow rules blocks nothing. A verdict of blocked names the matching rule, while a verdict of allowed with no rule means that nothing in the file applied to the path.
The simulator reads the file you paste, not the live one, so it describes the rules as written. It does not confirm how a crawler will behave, since robots.txt is a request that an agent may ignore. It also does not read noindex tags or response headers. For the live answer, run the single-URL test against the file the site serves.
The mistakes that deindex a site
The tool flags the patterns that quietly remove a site from search: a bare Disallow: / that blocks everything for its group, a disallow that matches stylesheet or script URLs so pages cannot render, and a sitemap that is itself blocked. It also warns when no Sitemap line is present. Each warning names the line responsible, so the cause is clear.
These matter because a blocked resource or a blocked site stops crawlers from reading pages, and a page that cannot be fetched cannot be indexed or rank. A sitemap line does not rescue a site that Disallow: / blocks, because the crawler will not fetch the listed URLs it is told to avoid. The warnings are conservative: they flag a pattern, and you judge whether it was intended.
Correct the file
Narrow broad rules to the paths you actually want excluded, and remove any disallow that matches CSS or JavaScript. Keep the rules for the paths that genuinely need protection, and keep the file readable so the next person can see what each group does. Add a Sitemap line that points at the live sitemap, and keep a copy of the previous file for comparison.
The asset-blocking check uses a set of common asset paths, so confirm which resources your pages really load before you decide a rule is safe. After editing, paste the new file and run the simulation again, then confirm the live result with robots test on a specific URL, since a harmless-looking rule can match more paths than expected. Test more than one URL before publishing.
When to use related checks
Use robots test for one live URL and one agent, robots check to review the live rules, and robots generator to write a starting file. The live check reads the file the site serves now, not the one pasted here, so the two are complements rather than duplicates. The single-URL test answers the narrower question for one live page, while this page covers the file as a whole.
For platform-specific guidance, see the Shopify robots.txt guide and the AI crawlers guide. Robots conflict looks for rules in a live file that contradict each other, which is where surprising blocks usually hide. The guide explains what each rule type does, and the conflict check compares rules that overlap so a broad rule cannot quietly catch a path you meant to allow.