What each one does
robots.txt tells crawlers which URLs they may fetch. It is a crawl directive. noindex, in a meta tag or an X-Robots-Tag header, tells a crawler that has fetched the page not to list it. It is an indexing directive.
The order is crawl first, then index. That single fact explains most of the confusion.
The failure: Disallow plus noindex
If you Disallow a URL and also put noindex on it, the crawler never fetches the page, so it never sees the noindex. The URL can remain in the index, listed without a snippet, because it is still known from links. The two directives together produce the opposite of what you wanted.
Use one or the other. If the goal is removal from search, allow crawling and use noindex. If the goal is to stop crawling a large pattern you never want indexed anyway, Disallow it and accept that any already-indexed URLs may linger.
Where noindex hides
The meta tag appears in the HTML. The X-Robots-Tag header does not, and it is set by a server, CDN or proxy. A staging configuration that adds the header often survives launch, and the page source looks completely clean. A check that reads response headers is the only way to see it.
The patterns worth blocking rather than noindexing
- Internal search results, which multiply with every query.
- Sorting and filtering parameters that duplicate a clean page.
- Cart, checkout and account paths, which no crawler should index.
- Admin paths, which should be protected rather than merely unindexed.
Settle it on your own site
The indexing check resolves robots.txt for the path and reads both forms of noindex, so it can tell you which of the two is acting and whether they are cancelling each other out.
- Indexing checkReads robots.txt, meta robots and the X-Robots-Tag header together.
- robots.txt, explainedHow wildcard and longest-match rules actually resolve.
- Robots.txt checkWhat your live file allows and blocks.
- AI crawler accessWhich AI agents your robots.txt refuses.
- robots.txt for ShopifyThe lines that matter, and what they block.