What ChatGPT-User does
Fetches a page when a ChatGPT user or custom GPT asks for it, rather than crawling automatically.
Because a person triggers these fetches, OpenAI documents that robots.txt rules may not apply to them in the same way. Check its current crawler documentation.
How to block ChatGPT-User in robots.txt
Add these two lines to the robots.txt file at the root of your domain: User-agent: ChatGPT-User and, on the next line, Disallow: /
To block only part of the site, replace / with a path such as /private/. Rules apply per host, so a subdomain needs its own robots.txt.
How to allow ChatGPT-User
If you have no rule for ChatGPT-User and no blanket User-agent: * Disallow: /, it is already allowed. To allow it explicitly while blocking other bots, add User-agent: ChatGPT-User followed by Allow: /
Check it worked
- Publish the updated robots.txt and open it in a browser at /robots.txt to confirm the live file has the rule.
- Run the free AI crawler check: it reads your live robots.txt and reports whether ChatGPT-User is allowed or blocked.
- Look for ChatGPT-User in your server or CDN logs over the following weeks to see whether requests stop.
robots.txt is a public request, not access control. OpenAI publishes its own documentation for ChatGPT-User; confirm the current token and behaviour there, since operators change them.
- Check which AI bots can read your siteTests your live robots.txt against every major AI agent.
- Block AI crawlersGenerate robots.txt rules for training bots, search bots or both.
- Every AI crawler user agentThe full list with operators and purposes.
- GPTBot vs ChatGPT-UserWhy OpenAI uses more than one agent.
Related explainers
- Why is my page not indexed
- robots.txt vs noindex
- Missing canonical
- Free technical SEO audit
- AI crawler user agents
- GPTBot user agent
- OAI-SearchBot user agent
- ClaudeBot user agent
- Claude-SearchBot user agent
- PerplexityBot user agent
- Google-Extended user agent
- Applebot-Extended user agent
- CCBot user agent
- Bytespider user agent
- Meta-ExternalAgent user agent
- Amazonbot user agent
- SEO MCP server
- 404 vs 410
- hreflang vs canonical
- What is a canonical tag