When a robots.txt rule blocks real pages, and how to find which rule

What the error means

Search Console reports pages excluded because robots.txt disallows their path. The pages are fine — the file is lying to the crawler about them, usually by accident: an over-broad Disallow prefix, a staging leftover, or an app that edited the template.

Why Google rejects it

robots.txt is prefix-matching and the longest matching rule wins. Disallow: /s does not mean "block /search" — it blocks every path starting with s, including products. A line written for staging or for one app’s endpoints can quietly swallow product paths, and because the pages return 200 in a browser, nothing looks wrong.

How to fix it

1. Read the live file at yoursite.com/robots.txt — not the template you remember. Apps rewrite it silently.
2. Test the affected URL against the rules: find every Disallow whose prefix matches, and remember the most specific rule wins.
3. Replace over-broad prefixes with full path segments (Disallow: /search/ not /s), and never leave a bare Disallow: / from a maintenance window.
4. On Shopify, keep /admin, /cart, /checkout blocked and /products/, /collections/, /pages/ explicitly allowed.
5. Recheck after any theme or app change — the file is edited by many hands.

Common causes

Broad disallow matches important paths

A rule such as Disallow: / blocks all crawling for the matching user-agent group. Less obvious prefixes can also catch public product or category paths. Robots rules use path matching, so read the exact rule and test the exact URL instead of assuming a similar-looking route is unaffected.

Specific user-agent group overrides the broad group

Robots.txt groups are evaluated by user-agent token. A rule under Googlebot can apply differently from a rule under User-agent: *. A broad Allow elsewhere may not cancel a more specific disallow for the same URL. Check the effective group for Googlebot and the URL path reported by Search Console.

Platform default was customized

A theme or SEO app may have added rules that hide filters, carts, or search results; a later edit can unintentionally include product pages. Shopify serves robots.txt from platform defaults unless a robots.txt.liquid template customizes it. Compare the live file with the intended rules and note which changes came from custom code.

CDN or server blocks the request too

Search Console's robots status points to the robots.txt rule, but removing that rule will not fix a separate firewall challenge or authentication requirement. A crawler may be allowed by robots.txt and still receive 403 or 429 from the origin. Check both robots evaluation and actual HTTP access.

robots.txt is on the wrong host or malformed

Robots.txt is scoped to a protocol, hostname, and port. A correct file on the www host does not control the bare host, or vice versa. A redirect, HTML error, or malformed line can also mean crawlers are applying unexpected rules. Fetch the file at the exact origin serving the blocked URL.

Fix it on your platform

On Shopify

1. Open https://your-domain.example/robots.txt on the primary storefront domain and locate the rule matching the exact product or collection path. Shopify's default rules intentionally block some utility and duplicate URL patterns.
2. If a custom robots.txt.liquid exists, go to Online Store > Themes > Edit code and inspect the published theme's template. Make a narrow change to the relevant directive; do not replace Shopify's full default rules with a blanket allow or disallow.
3. Check whether an app or external CDN has additional bot rules. A robots.txt Allow cannot bypass a firewall challenge, and disabling Shopify's default exclusions wholesale can expose low-value URLs to crawling.
4. Save the published theme change, fetch /robots.txt again, and test the exact path using Google's robots.txt testing or URL Inspection tools.

On WooCommerce

1. Fetch /robots.txt from the exact public hostname and check for a Disallow rule that matches the affected path. WordPress may serve a virtual robots.txt when no physical file exists; a physical file or SEO plugin can change the output.
2. Inspect WordPress SEO plugin settings and any robots.txt editor for custom directives. Remove only the unintended path restriction and preserve blocks for private admin or duplicate utility routes where appropriate.
3. If a physical robots.txt file is present, edit it through the host's file manager or deployment process and check that a plugin or cache is not serving a different version.
4. Review security plugins, hosting firewall, and CDN rules separately. Check access logs for Googlebot response codes because robots.txt permission does not itself allow a request through those layers.

On other platforms

1. Use the platform's SEO, marketing, or developer settings to locate its robots.txt control. Wix, BigCommerce, and Squarespace may expose different customization options; verify the live file rather than relying on the editor preview.
2. On custom hosting, edit the robots.txt file at the web root for the same protocol and hostname as the affected URL. Confirm deployment did not publish the file to a subdirectory.
3. Test the full URL path against the matching user-agent group. Preserve intentional blocks for account, checkout, and internal search pages unless there is a specific reason to change them.
4. Check the CDN or WAF access logs for the crawler separately, especially when the live robots file allows the URL but Search Console still cannot fetch it.

How to confirm it is fixed

1. Fetch robots.txt on the same host and protocol as the affected URL, then identify the applicable user-agent group and the rule that matches its path.
2. Use Search Console's robots.txt report or URL Inspection live test for the exact affected URL. Confirm crawl is allowed and that the page itself returns an accessible response.
3. Check other important URLs in the same path family to ensure the change did not open private routes or continue blocking product and category pages unintentionally.
4. Run the robots test against the URL and crawler token, and review CDN or server logs for separate access denials.
5. After Google can crawl the page, use URL Inspection to request recrawling where appropriate and monitor the Page indexing report for a new crawl result.

A worked example

Hypothetical case: a Shopify merchant wants to keep filtered collections out of crawl and adds a custom rule for /products to the robots.txt.liquid template after copying an example from a forum. Search Console soon reports product URLs as blocked, while shoppers can still buy them. The owner checks the live robots.txt, finds that the custom group disallows the entire product path, and compares it with Shopify's default rules. They remove the overbroad directive but retain the intended filter exclusions, publish the theme, and test a product URL against Googlebot. URL Inspection then confirms Google can fetch the page; indexing status updates after a later crawl.

Other errors like this