ShipworkSite inspection
Checks

Crawlability

PaginationAre paginated pages set up right?Internal searchIs site search eating crawl budget?URL parametersAre parameters creating duplicates?Log analyserWhat does Googlebot actually crawl?Robots builderNeed a robots.txt file?Soft 404sAre pages “not found” but returning 200?JavaScriptCan crawlers see it without running JS?FreshnessIs my site quietly going stale?Robots.txtDoes robots.txt say what I think?

Indexing

Sitemap lastmodAre my sitemap dates valid?SitemapIs my sitemap actually fetchable?IndexingIs Google allowed to index my pages?CanonicalsIs Google indexing the wrong URL?Index signalsDo sitemap and index signals agree?Sitemap generatorNeed a sitemap.xml for my site?Bulk URL statusWhere does each of these URLs really land?Redirect builderNeed the redirect rules?RedirectsAre my URLs answering directly, over HTTPS?

On-page

Keyword ideasWhat are people searching for?SERP previewHow does my page look in Google?Content qualityAre your pages too thin or too alike?Image weightAre images slowing the page?CannibalizationAre pages competing with each other?On-page checkDoes the page use its target phrase?ImagesAre my images accessible and loading?Meta tag builderWhat should my title and share tags say?DuplicatesDo my pages compete for one query?

Links

Orphan pagesWhich pages can no link reach?Link graphHow deep do your pages sit?Anchor textDo links say what they point at?Outbound linksAre external links still alive?Broken linksAre internal links sending visitors nowhere?

Structured data

Rich resultsIs my markup eligible for a rich result?Structured dataIs my product schema valid?Schema coverageDo my key pages carry structured data?Social previewHow does my page look when shared?Schema vs pageDoes markup match the page price?Schema builderNeed valid JSON-LD?

International

Hreflang sitemapDo page and sitemap hreflang agree?HreflangDo my language versions link back?Hreflang builderNeed the hreflang tags?Page DoctorWhy is this page not doing well?Redirect planMy site has dead links — what redirects do I write?Sitemap diffIs anything missing from my sitemap?Robots simulatorWhat does my robots.txt actually block?Can Googlebot?Can Googlebot fetch this URL?
Every error, explainedGuides
Pricing Learn Error guides Run free audit

+1 free check

Welcome to Shipwork

Sign in free and your next check this month is on us.

Continue with GoogleContinue with email

We email you a 6-digit code. No password.

No card needed. Reports and watches follow you to any device.

llms.txt vs robots.txt

robots.txt tells crawlers what they may access. llms.txt tells AI tools what your site is and which pages matter. Only robots.txt can ask a crawler to stay out; llms.txt cannot block anything.

Published

Side by side

robots.txtllms.txt
PurposeAccess rules for crawlersA guide to the site for AI readers
StatusLong-established standard (RFC 9309)A proposal, read by some tools
FormatUser-agent, Allow and Disallow linesMarkdown: title, summary, link lists
Can it block AI crawlers?Yes, by user agentNo
Location/robots.txt/llms.txt

Using both

They answer different questions, so most sites that care about AI search use both. robots.txt decides which AI agents may read the site, for example blocking training crawlers while allowing AI search. llms.txt then points the allowed agents at the pages worth reading. If robots.txt blocks an agent, your llms.txt does not help with that agent.

Common mistakes

  • Trying to block AI in llms.txt. It has no access rules. Use robots.txt; the block AI crawlers generator writes them.
  • Blocking all AI agents, then adding llms.txt. The agents you blocked will not read it.
  • Listing pages in llms.txt that robots.txt disallows. Keep the two consistent.
Find out if this is happening to you

Shipwork reads robots.txt for every major AI agent and checks whether /llms.txt is present. Free, no account, no signup. Paste your store address.

Check my AI crawler access

Questions

Does llms.txt replace robots.txt?
No. robots.txt sets access rules; llms.txt only describes the site.
Can llms.txt block GPTBot?
No. Only robots.txt (or server-level blocking) can. Add User-agent: GPTBot with Disallow: / to robots.txt.
Do I need both?
robots.txt, yes, for any site. llms.txt is optional and cheap to add if you want AI tools to find your key pages.

Keep reading