How a page Google may not crawl still ends up in search, and how to decide what to do

What the error means

robots.txt tells Google not to crawl the URL, but Google indexed it anyway, usually without reading the page. It may appear in results with no description or a note that no information is available.

Why Google rejects it

robots.txt controls crawling, not indexing. If other pages link to a blocked URL, Google can index the address from those links alone. Because it cannot crawl the page, it also cannot see a noindex tag on it.

How to fix it

1. Decide what you want for each URL: indexed, or out of search.
2. If it should be indexed, remove or narrow the robots.txt rule that blocks it.
3. If it should stay out of search, allow crawling and add a noindex meta tag or X-Robots-Tag header. Google has to crawl the page to see the noindex.
4. After Google has recrawled and dropped the page, you can block it in robots.txt again if crawl load matters.
5. Remove links to these URLs from your sitemap.

Common causes

An external link exposed the URL

Google can discover a URL from a link even when robots.txt prevents it from fetching the page. It may index the URL based on external signals, but cannot read the blocked page content, so its search result can have limited information.

The page was indexed before the block

Google may have crawled and indexed the page before a later robots rule was added. The block stops subsequent fetches; it does not automatically remove the existing result from the index.

The page should remain public in Search

A broad disallow can accidentally cover product, category, or editorial pages. If you want the content crawled and indexed, remove or narrow the matching robots rule instead of trying to hide it with a conflicting directive.

The page should be removed from Search

Robots.txt controls crawling, not reliable deindexing. If you keep the block in place, Google cannot fetch a page-level noindex directive. Use a crawlable noindex page, a password requirement, or remove the content depending on the goal.

Fix it on your platform

On Shopify

1. Open the live /robots.txt on the same domain as the affected URL and locate the rule matching the product or page path. Shopify's defaults block some utility routes; distinguish those from a custom broad rule.
2. If the rule is custom, review robots.txt.liquid in the published theme. Narrow the directive only if the page should be crawled; do not remove all default exclusions without checking their purpose.
3. For a page that should not appear in search, allow Google to crawl it and apply a supported noindex setting where available, or protect/remove the content. A robots block alone is not deindexing.
4. Check app, CDN, and storefront password settings separately; robots permission does not bypass access control.

On WooCommerce

1. Inspect the live robots.txt file and the rule that matches the exact URL. Check both a physical file and any SEO plugin robots editor.
2. If the page should be searchable, remove the unintended disallow and confirm no other group blocks Googlebot. If it should be removed, use the SEO plugin's noindex control after allowing crawl access.
3. Check Search Console URL Inspection for whether Google can fetch the page now and whether a previous indexed version remains.
4. Avoid adding a noindex directive behind a disallow; Google must crawl the URL to see the directive.

On other platforms

1. Use the platform's robots settings or help documentation to change the exact path rule; verify the live file after publishing.
2. For a public page meant to rank, permit crawling and maintain normal indexability signals. For a page to exclude, use noindex or access protection rather than robots.txt alone.
3. On custom hosting, check CDN and origin robots output for the exact hostname, because each host has its own robots scope.
4. If the URL was indexed before the rule was added, allow crawling again if you need Google to process a noindex or removal change.

How to confirm it is fixed

1. Use URL Inspection to confirm the exact URL and verify whether it is blocked in the indexed version and live test.
2. Test the URL against the live robots file for Googlebot and confirm the resulting rule; check that the file is served on the page's actual host.
3. Run the robots test for the URL, then inspect the page's noindex and access state according to whether you want it indexed.
4. If you changed a rule, allow Google to recrawl before assuming the existing search result has updated.

A worked example

Hypothetical case: a retailer blocks /products/ in robots.txt while trying to reduce crawler requests to product filters. A review site links directly to a popular product, and Google indexes the URL from that link even though it cannot fetch the page. The result appears with a bare URL and little snippet information. The retailer decides the product should rank, narrows the robots rule to the filter pattern, and confirms Googlebot can fetch the product. They inspect the live URL and wait for Google to recrawl; the existing indexed result does not disappear merely because the rule changed.

Other errors like this