The template that works
User-agent: * Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /carts Disallow: /account Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Sitemap: https://yourstore.com/sitemap.xml
The shape is deliberate: transactional and account paths closed, content paths open, sitemap declared with the exact live URL. Replace the domain, keep the shape.
What stays disallowed
| Path | Why closed |
|---|---|
| /admin, /account | No crawler business behind logins; login pages dilute demand |
| /cart, /carts, /checkout* | Transactional dead-ends that multiply URL space |
| /orders | Private order-status pages with zero index value |
What must stay crawlable
Products, collections, pages and blogs are the indexable surface of the store — each needs an explicit Allow so no broader rule can accidentally swallow it. The Sitemap line must name the exact root URL Google should fetch; a stale domain here is one of the top “could not fetch” causes. Anything AI-specific lives in named-agent groups alongside this base, covered in the AI crawler guide.
The mistakes that hurt
- Disallow: / under User-agent: * — total blackout, usually left over from a staging or maintenance window. Check first, always.
- Blocking /products/ or /collections/ — sometimes added by SEO apps during setup. Products vanish from the index while the store looks untouched.
- Over-broad prefixes — Disallow: /s blocks everything containing /s. Prefer full path segments with trailing slashes.
- Edits nobody remembers — apps and copied snippets rewrite the template silently. Re-read the live file after every theme or app change.
Fetch /robots.txt fresh and test your top product and collection URLs against it. Memory of what the file “should” contain is how stores stay blocked for months.
Deeper: robots.txt for AI crawlers · sitemap pillar · Related guide: could not fetch
Shipwork fetches your robots.txt and sitemap together from outside your network — verifying the sitemap reference resolves and nothing crawlable is blocked. Free, no account, no signup — paste your store address.
Check my sitemapQuestions
- Where is the Shopify robots.txt file?
- Served at yourstore.com/robots.txt, generated from a template you can edit. Read the live file, not the template — apps can change the output.
- Should I block the search (/search) pages?
- Yes — internal search results add URL space without index value. Disallow /search while keeping products and collections open.
- Why does my sitemap line matter so much?
- It tells crawlers exactly which file to fetch. A stale domain or path here is a top-three “could not fetch” cause.
- How do AI agent rules fit in?
- Named-agent groups sit alongside the general rules; the most specific group wins per agent. The AI crawler guide covers the nine agents that matter.
Keep reading