What Bytespider does
ByteDance’s crawler, widely reported to collect data for its AI models.
Some site owners report it does not consistently follow robots.txt. If it matters, check your server logs and consider blocking it at the firewall as well.
How to block Bytespider in robots.txt
Add these two lines to the robots.txt file at the root of your domain: User-agent: Bytespider and, on the next line, Disallow: /
To block only part of the site, replace / with a path such as /private/. Rules apply per host, so a subdomain needs its own robots.txt.
How to allow Bytespider
If you have no rule for Bytespider and no blanket User-agent: * Disallow: /, it is already allowed. To allow it explicitly while blocking other bots, add User-agent: Bytespider followed by Allow: /
Check it worked
- Publish the updated robots.txt and open it in a browser at /robots.txt to confirm the live file has the rule.
- Run the free AI crawler check: it reads your live robots.txt and reports whether Bytespider is allowed or blocked.
- Look for Bytespider in your server or CDN logs over the following weeks to see whether requests stop.
robots.txt is a public request, not access control. ByteDance publishes its own documentation for Bytespider; confirm the current token and behaviour there, since operators change them.
- Check which AI bots can read your siteTests your live robots.txt against every major AI agent.
- Block AI crawlersGenerate robots.txt rules for training bots, search bots or both.
- Every AI crawler user agentThe full list with operators and purposes.
- GPTBot vs ChatGPT-UserWhy OpenAI uses more than one agent.
Related explainers
- Why is my page not indexed
- robots.txt vs noindex
- Missing canonical
- Free technical SEO audit
- AI crawler user agents
- GPTBot user agent
- OAI-SearchBot user agent
- ChatGPT-User user agent
- ClaudeBot user agent
- Claude-SearchBot user agent
- PerplexityBot user agent
- Google-Extended user agent
- Applebot-Extended user agent
- CCBot user agent
- Meta-ExternalAgent user agent
- Amazonbot user agent
- SEO MCP server
- 404 vs 410
- hreflang vs canonical
- What is a canonical tag