Is your internal search eating your crawl budget?

A site search creates a URL for every query. Left crawlable, those pages multiply without limit and pull the crawler away from the pages you want indexed. One robots.txt line usually fixes it.

Looks for internal search URLs and whether robots.txt and noindex block them. Saved to your account if you are signed in to a paid plan.

How this check works

Shipwork looks for internal search URLs in the pages it checks, resolves your robots.txt for the /search path, and checks whether any search page it found is marked noindex. It reports an unblocked search, or a search page that could still be indexed.

What the internal search indexing check finds

The internal search indexing check looks for three signs of search: links on the site that carry query keys such as s, q, query, search or keyword, links to a path that begins with /search, and sampled pages whose own URL is a search path. If it finds none, it says so, and if the home page shows a search box or dialog it notes that search may be client-side.

Where search exists, the check reads robots rules for the /search path and reports whether the wildcard agent is disallowed there. It also inspects any sampled search page for a noindex directive. The free run reads a sample of up to 20 pages and the robots file, the same files a crawler reads.

How to read an internal search result

A warning that search is not blocked means your robots rules do not disallow the /search path for all agents. That is a crawl-budget risk rather than a ranking penalty. A clean result means it is disallowed. A separate warning means a sampled search page is missing noindex, so it could still be indexed even when crawling is discouraged.

The check cannot enumerate every query URL a crawler might follow, and it does not prove that a search engine respected your rules. Client-side search that produces no distinct URLs is not a crawl-budget problem at all. Read the finding as a statement about what the public files allow, then verify in a live crawl log if the volume matters.

Common causes and why they matter

Most sites add search once and never think about it again, so the /search path is left open. Each query then becomes a URL, and a crawler that follows internal search links can generate a long tail of thin, near-identical result pages. Faceted navigation and sort links behave the same way.

Crawl attention is finite. Time a crawler spends on endless search-result URLs is time it does not spend on products, articles and category pages. Those thin result pages can also be indexed and compete with the pages they merely list. Blocking search is one of the cheapest crawl-budget improvements a site can make.

How to fix internal search indexing

Add a disallow rule for the search path and for the query keys your platform uses, for example a line that disallows /search and any address beginning with ?s= or ?q=. Then make search result pages noindex as well, so they cannot appear in results even if they are linked. Keep the search box working for visitors; this only changes what crawlers are invited to read.

If your search is entirely client-side and never produces a distinct URL, there is nothing to block, and the check will simply report that it found no crawlable search pages. After editing robots rules, re-run the check and confirm the path now reads as disallowed, and spot-check a real query URL to be sure it is not indexable.

When to use related checks

Use robots check for the rest of your robots rules, including whether important paths are blocked by mistake. Use URL parameters when query strings beyond search are creating duplicate addresses.

The crawl budget guide and the robots rules guide explain how search and filters consume crawl attention. Internal search is one source of wasted URLs; parameters and faceted navigation are others, and each has its own check.

Questions

Why block internal search?

Every query creates a URL. Crawlers follow them, spend their budget there instead of on products or articles, and can index thin, duplicate result pages.

How do I block it?

Add "Disallow: /search" to robots.txt, and make search pages noindex so they cannot appear in results even if linked.

Should the search page itself be blocked?

Usually yes: both the /search path and query patterns like ?s= or ?q=. Keep it usable for visitors, just not for crawlers.

How do I block internal search from Google?

Disallow the search path and its query keys in robots rules, for example /search and addresses beginning with ?s= or ?q=, then set search result pages to noindex. Blocking crawling and preventing indexing are separate steps, so do both where the platform allows it.

Is my site search bad for SEO?

Search itself is fine and useful for visitors. The risk is that each query creates a URL a crawler can follow, spending crawl attention on thin result pages instead of real content. Blocking crawl access to those URLs is a common fix; blocking your users is not.

Should search result pages be noindex?

Yes, usually. A noindex directive stops them appearing in results even if something links to them. It works best alongside a robots disallow, which stops them consuming crawl attention in the first place. A page that is both disallowed and noindex is a clear signal to leave it out.

Does the internal search check read my robots file?

Yes. It reads the public robots rules and checks whether the wildcard agent is disallowed for the search path, then inspects any search page it sampled for a noindex directive. It reads these as a crawler would, from outside your network.

What if my search is client-side only?

If search results never create distinct URLs, there is no crawl-budget problem and nothing to block. The check may note that a search input or dialog exists while finding no crawlable search pages. Confirm whether your search changes the URL before adding rules.

Keep reading

What this reads

Reads public pages from outside your network, the way a crawler does.
No signup, and nothing is stored on the free checks.
One of 83 checks across discovery, content, markup, trust and catalogue data.

83checks

0signups needed

0stored on free checks

Cool, just ran an audit and got some insights! Let’s see if I can fix them :)

Tenta Dev, @DevTenta

Whoah! this site is crazy, nothing to roast at all

Ainreek, @AinreekAI

cool as heck, tried it on my blog and the insights it gave are helpful and not bluff.

Refaat 🍉 (Mouse), @1Mouse__