Crawl beyond the first few pages
A free check reads a handful of pages. The deep crawl reads up to 500 sitemap URLs and reports problems that only show up at scale: dead pages, noindex entries, duplicate titles and thin content.
Crawls up to 500 sitemap URLs, subject to a 90-second time budget. This is the resource-heavy check; nothing is stored.
Not connected. Sign in and Connect Search Console to read Google's own verdict for this page. The check below still runs from public signals without it.
How this check works
Shipwork reads every URL in your sitemap (up to the limit) and fetches each one, then aggregates: dead or unreachable pages, sitemap URLs that redirect, pages set to noindex, missing and duplicate titles, missing descriptions and thin content. It is time-budgeted and reports how far it got.
What the deep crawl site checker reads
The deep crawl starts from the URLs in your XML sitemap. It reads the sitemap at your origin, takes up to 500 URLs and fetches them six at a time. For each URL it records the HTTP status, the title, the meta description, whether the page is set to noindex, its canonical link and a word count. It is the same signal set a small sample reads, run across the catalogue rather than the first few pages.
The limit is 500 URLs and the run has a 90-second budget, so a slow site may finish fewer and the result reports how far it got. It requests raw HTML and does not run JavaScript, so content a browser would build is not counted. It does not build a link graph, fetch images or read structured data; those are separate checks.
How to read a deep crawl result
A clean run means the fetched URLs returned pages with titles, descriptions and enough text, and none were dead or marked noindex. It applies to the URLs actually fetched within the budget, which may be fewer than the whole sitemap. The per-page list shows the status of every URL it read, so you see the evidence rather than only a count.
Each finding names a count and a fix. A dead-page finding counts statuses that returned nothing or an error. A redirect finding counts sitemap URLs that answer 300 to 399, which usually means the sitemap lists an old address. A thin-page finding counts pages under 200 words. None of these prove a ranking problem on their own; they are maintenance signals found at scale.
Common causes and why they matter for search
Deep crawls surface problems a sample cannot: a deleted product still listed in the sitemap, a paginated URL that redirects, or a template that emits the same title on hundreds of pages. At scale those become crawl waste and thin duplicate pages. A page marked noindex but still listed in the sitemap sends search engines a contradictory instruction.
The thin-content count uses a simple word threshold, so a page with a short but complete answer can be counted as thin. Treat it as a shortlist to review, not a verdict. A dead URL in the sitemap matters because the sitemap is meant to list indexable pages; repeated errors there reduce a crawler's confidence in the file.
How to fix what the deep crawl finds
Work through the findings in the order they cost: dead pages first, then noindex URLs in the sitemap, then duplicate and missing titles, then descriptions, redirects and thin pages. Remove dead and redirected URLs from the sitemap or point them at the final page. Give every indexable page a unique title and fill in descriptions where they matter.
On Shopify the sitemap is generated, so fix the underlying product, page or collection rather than editing the file. In WooCommerce, check the SEO plugin's sitemap settings and the product status. After changes, rerun the crawl, since a generated sitemap can take time to reflect edits. Expand or merge genuinely thin pages instead of padding them.
When to use related deep crawl checks
Use the free sitemap check first when the sitemap itself may be broken, since the deep crawl needs one to work. Use the site audit for a quick sample on any site, and the indexability check when one URL should not be indexed.
For crawl efficiency on a large storefront, read crawl budget on Shopify. The deep crawl is a paid, resource-heavy run, so keep it for the moments you need the whole picture rather than for every check.
Questions
How is this different from the free audit?
How many URLs does it crawl?
Is it heavy?
How many pages does the deep crawl check?
Does the deep crawl run JavaScript?
Why does the deep crawl say no sitemap was found?
What counts as a thin page in the crawl?
Is the deep crawl free?
Keep reading
What this reads
- Reads every URL your sitemap declares, not a sample.
- Reports dead pages, redirects, noindex entries and duplicate titles across the site.
- One crawl, priced by the size of your sitemap.
Cool, just ran an audit and got some insights! Let’s see if I can fix them :)
Whoah! this site is crazy, nothing to roast at all
cool as heck, tried it on my blog and the insights it gave are helpful and not bluff.