Side by side
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Access rules for crawlers | A guide to the site for AI readers |
| Status | Long-established standard (RFC 9309) | A proposal, read by some tools |
| Format | User-agent, Allow and Disallow lines | Markdown: title, summary, link lists |
| Can it block AI crawlers? | Yes, by user agent | No |
| Location | /robots.txt | /llms.txt |
Using both
They answer different questions, so most sites that care about AI search use both. robots.txt decides which AI agents may read the site, for example blocking training crawlers while allowing AI search. llms.txt then points the allowed agents at the pages worth reading. If robots.txt blocks an agent, your llms.txt does not help with that agent.
Common mistakes
- Trying to block AI in llms.txt. It has no access rules. Use robots.txt; the block AI crawlers generator writes them.
- Blocking all AI agents, then adding llms.txt. The agents you blocked will not read it.
- Listing pages in llms.txt that robots.txt disallows. Keep the two consistent.
Shipwork reads robots.txt for every major AI agent and checks whether /llms.txt is present. Free, no account, no signup. Paste your store address.
Check my AI crawler accessQuestions
Does llms.txt replace robots.txt?
Can llms.txt block GPTBot?
Do I need both?
Keep reading