What "not indexed" actually means
Search Console's page-report statuses form a pipeline, and each failure point has a different fix. Discovered - currently not indexed means Google knows the URL exists but has not scheduled a crawl — usually a crawl-budget or quality signal problem. Crawled - currently not indexed means Google fetched the page and decided not to index it — thin content, duplicate, or a quality judgment. Excluded by noindex tag and Blocked by robots.txt are instructions you sent, deliberately or not. The first two are Google's decisions; the last two are yours to fix, and they are the ones this article is about, because they are the ones nobody remembers setting.
| Status | Who decided | Where to look |
|---|---|---|
| Discovered - not indexed | Google (crawl scheduling) | Crawl budget, internal links, site quality |
| Crawled - not indexed | Google (quality) | Thin or duplicate content, canonicals |
| Excluded by noindex | You (page instruction) | Meta robots tag, CMS settings, SEO plugin |
| Blocked by robots.txt | You (file instruction) | robots.txt rules, hosting defaults |
The four silent causes
- A noindex meta tag left by a plugin or template. SEO plugins, staging-to-production migrations and " discourage search engines" checkboxes left flipped are the classic sources. The page renders perfectly; the tag sits one line into the head where nobody looks.
- An X-Robots-Tag HTTP header. The same instruction delivered in the response header instead of the HTML — set by a CDN rule, a hosting default or a security plugin. View-source never shows it; only a header inspection does.
- A robots.txt rule that swallows real pages. A staging leftover like Disallow: / under a named agent, or an over-broad prefix that catches /products along with /search. Shopify stores get extra risk from apps that edit the template.
- The page errors for crawlers. A 403 from bot protection or a 500 on a cold code path reads to Google as an unusable page, which is indistinguishable from an instruction not to index.
A noindex tag, a response header and a crawler block are all invisible to a logged-in visitor on their own network. Every one of them is obvious to a cold outside request — which is exactly what the check does.
"Discovered - currently not indexed"
This status deserves its own paragraph because the advice around it is mostly wrong. It does not mean Google dislikes the page; it means the URL is known but the crawl is not scheduled yet. On a new or small site it is usually crawl-budget triage: Google allocates attention based on how many links point at a URL and how the whole site performs. The practical fixes are internal linking (orphans and deep pages suffer most), pruning near-empty pages out of the crawl, and patience — not resubmitting the same URL repeatedly, which changes nothing.
How to find your blocker in a minute
Search Console tells you the status per URL but not the mechanism. The fast path: run the free indexing check — paste your address and Shipwork fetches a sample of your key pages from outside your network, reading the same signals a crawler reads first: the raw HTML meta robots, the X-Robots-Tag header, robots.txt rules, and the response status each page returns. It reports which pages are told not to index, which are blocked, and which error out — so you know whether to fix a tag, a header, the file or the firewall. Free, no signup, nothing stored.
Related: Shopify robots.txt, line by line · crawl budget on Shopify · sitemap fetch errors
Shipwork fetches your key pages from outside your network and reads the exact signals a crawler reads first — meta robots, X-Robots-Tag, robots.txt rules and response status — then names which page is blocked by what. Free, no account, no signup — paste your store address.
Check my indexingQuestions
- How do I check if a page is indexed?
- Search for the exact URL, or use the URL Inspection tool in Search Console. Both tell you the current state; neither tells you why — that is what the signals in the page source and headers are for.
- What is the difference between crawled and discovered, not indexed?
- Discovered means Google knows the URL but has not crawled it yet — a scheduling and budget signal. Crawled means Google read the page and chose not to index it — a quality or duplicate-content judgment.
- Does noindex pass PageRank?
- A noindex page is eventually dropped from the index, and its links stop counting the way links to indexed pages do. Remove noindex from pages you want ranking, or remove the pages entirely.
- Can robots.txt cause "Discovered - not indexed"?
- Not directly — robots.txt prevents crawling, which shows as "Blocked by robots.txt" instead. But heavy blocking elsewhere on the site can shape crawl-budget decisions overall.
Keep reading