What Screaming Frog does well
Screaming Frog is a mature desktop crawler. It crawls large sites in depth, exports every URL it found, integrates with search APIs and log files, and gives you a level of detail and configuration a web tool does not. The free version covers up to five hundred URLs. If you need a full-site export with per-URL data, it is the right tool.
Where a desktop crawler is structurally limited
When you run a crawler on your own machine, you see your site from your own network, often warmed and sometimes whitelisted. The failure mode that costs merchants the most is exactly that gap: a page that loads for you and returns a 403 or a 503 to Google. A request from a different network is the only way to reproduce it.
The same limit applies to checks that are not about fetching a page at all: whether AI agents are allowed to read your site, what real users experience on mobile, and whether a Merchant Center field is present. Those need a server view, not a local one.
What Shipwork does instead
- Runs from outside your network, so reachability and bot-protection failures are reproducible rather than hidden.
- Checks AI crawler access, real-user Core Web Vitals and Merchant Center product data, which a local HTML crawl does not cover.
- Needs no install, no licence and no crawl budget of your own, and gives one page per finding with the specific fix.
- Checks a set of pages rather than exporting, so it is faster and cheaper but less exhaustive than a full crawl.
Which should you use?
If you need a complete URL-level export, use Screaming Frog. If your question is can crawlers reach and read my site, and is anything quietly blocking them, run the free audit here first and use a desktop crawler to go deeper once you know where to look.
- Run the free auditOne crawl, every applicable check, from outside your network.
- Every check, explainedThe full list, each with what it inspects and the fix.
- Product page checkWhat your product URLs return to an outside crawler.
- AI crawler accessWhether ChatGPT, Claude and others may read your store.
- Shipwork vs Screaming FrogInstall a crawler, or just paste a URL?