ShipworkSite inspection
Checks
Every error, explainedGuides →
Pricing Learn Guides

robots.txt blocking vs noindex: which do you actually need?

People reach for both to hide a page, then wonder why it stays in search. They act at different stages, and combining them in the wrong order cancels the effect you wanted.

Resolves robots.txt for your pages and reads both a meta robots tag and the X-Robots-Tag header, from outside your network. Nothing is stored.

What each one does

robots.txt tells crawlers which URLs they may fetch. It is a crawl directive. noindex, in a meta tag or an X-Robots-Tag header, tells a crawler that has fetched the page not to list it. It is an indexing directive.

The order is crawl first, then index. That single fact explains most of the confusion.

The failure: Disallow plus noindex

If you Disallow a URL and also put noindex on it, the crawler never fetches the page, so it never sees the noindex. The URL can remain in the index, listed without a snippet, because it is still known from links. The two directives together produce the opposite of what you wanted.

Use one or the other. If the goal is removal from search, allow crawling and use noindex. If the goal is to stop crawling a large pattern you never want indexed anyway, Disallow it and accept that any already-indexed URLs may linger.

Where noindex hides

The meta tag appears in the HTML. The X-Robots-Tag header does not, and it is set by a server, CDN or proxy. A staging configuration that adds the header often survives launch, and the page source looks completely clean. A check that reads response headers is the only way to see it.

The patterns worth blocking rather than noindexing

  • Internal search results, which multiply with every query.
  • Sorting and filtering parameters that duplicate a clean page.
  • Cart, checkout and account paths, which no crawler should index.
  • Admin paths, which should be protected rather than merely unindexed.

Settle it on your own site

The indexing check resolves robots.txt for the path and reads both forms of noindex, so it can tell you which of the two is acting and whether they are cancelling each other out.

Questions

Does noindex remove a page from search immediately?
No. The crawler has to fetch the page again to see the noindex, so removal happens on the next crawl, which can take days or weeks.
Is it bad to Disallow a URL that is already indexed?
It can keep the URL listed without a snippet. If your goal is removal, allow crawling and use noindex instead.
Where do I put noindex?
In a meta robots tag in the HTML, or in an X-Robots-Tag response header. The header also applies to non-HTML files where a meta tag cannot be used.
Should I block search pages or noindex them?
Usually both patterns are unnecessary. Choose one. For internal search, Disallow is often enough, because you never want those URLs crawled. For pages already indexed that you want gone, noindex is the one that works.