Is your internal search eating your crawl budget?
A site search creates a URL for every query. Left crawlable, those pages multiply without limit and pull the crawler away from the pages you want indexed. One robots.txt line usually fixes it.
Looks for internal search URLs and whether robots.txt and noindex block them. Nothing is stored.
How this check works
Shipwork looks for internal search URLs in the pages it samples, resolves your robots.txt for the /search path, and checks whether any search page it found is marked noindex. It reports an unblocked search, or a search page that could still be indexed.
Questions
- Why block internal search?
- Every query creates a URL. Crawlers follow them, spend their budget there instead of on products or articles, and can index thin, duplicate result pages.
- How do I block it?
- Add "Disallow: /search" to robots.txt, and make search pages noindex so they cannot appear in results even if linked.
- Should the search page itself be blocked?
- Usually yes — both the /search path and query patterns like ?s= or ?q=. Keep it usable for visitors, just not for crawlers.