What this check inspects
Shipwork reads robots.txt for named AI crawler groups and a Content-Signal line, asks the home page for Markdown with an Accept: text/markdown header, and looks for the sitemap. It then reads the well-known discovery documents: the API catalog, auth.md, OAuth and protected-resource metadata, the A2A agent card, the agent-skills index, the MCP server card and a Web Bot Auth key directory, checks the home page for a discovery Link header and in-browser WebMCP tools, and notes the commerce documents (ACP, UCP, MPP). Every read is passive: nothing is posted or registered.
What a failure means
Level 1 is for every site: a missing robots.txt or sitemap is worth fixing, and missing Content Signals, named AI crawler rules or Markdown for agents are notes, because they are newer standards. Levels 2 and 3 only matter for a site that offers an API or an agent, and commerce only for one that wants agents to buy from it, so a missing item there is never a fault.
How to fix it
- Add a Content-Signal line to robots.txt stating whether AI may use your content for search, answers and training.
- Name the AI crawlers you care about (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) in their own robots.txt groups.
- Serve a Markdown copy of pages to requests that ask for text/markdown; Cloudflare can do this at the edge.
- If you have an API, publish /.well-known/api-catalog, a Link header pointing to it, and an auth.md explaining access.
A typical failure, worked through
The setupA SaaS site allows every crawler through the wildcard rule and has a public API documented only on a human docs page.
What the check reportsThe check passes robots.txt and the sitemap, notes that no AI crawler is named and no Content Signals are declared, and shows discovery at 0 of 3 because nothing tells an agent the API exists.
The pointBeing crawlable is not the same as being usable. The discovery documents are small files, and they are what an assistant reads to decide it can act on your site.