How to block GPTBot without blocking every OpenAI crawler

Block GPTBot in robots.txt

For a standards-based request to exclude GPTBot from the entire host, add this group to the root /robots.txt:

User-agent: GPTBot
Disallow: /

This is OpenAI’s documented opt-out signal for content that may be used to train its generative AI foundation models. It does not block other OpenAI agents. If you want to remain eligible for ChatGPT Search, leave OAI-SearchBot allowed. These controls are independent, according to OpenAI’s crawler documentation.

A robots.txt directive is not authentication and only applies to crawlers that respect it. Use it for crawler preferences, not for secrets or restricted content. Check that the file is available at the root of every hostname you intend to cover.

Filter GPTBot requests at Cloudflare

If you need an enforcement layer for requests reaching your Cloudflare zone, create a WAF custom rule matching the request’s User-Agent header for the GPTBot token, then choose a block action. In the Cloudflare dashboard, the rule builder is under Security → WAF → Custom rules; exact labels can vary as the dashboard changes. Scope the rule to the hostname(s) you intend to protect and test it against logs before broad deployment.

A user-agent match is spoofable: any client can send that header, and a crawler can change its request string. Cloudflare’s verified bot signals or published provider IP ranges can provide additional confidence where available, but ranges and product features change. Follow Cloudflare’s current custom rules documentation and the provider’s current IP guidance rather than hard-coding a copied list.

Keep the robots.txt group even if you add a WAF rule. The file communicates your preference to compliant crawlers; the firewall is the request-level enforcement mechanism. Neither substitutes for access control on private pages.

Filter with Nginx

For an Nginx site where you deliberately want to reject matching requests, a server-level conditional can return 403:

map $http_user_agent $is_gptbot {
    default 0;
    ~*GPTBot 1;
}

server {
    if ($is_gptbot) { return 403; }
    # existing server configuration
}

Place the map in the http context, not inside the server block; validate configuration with nginx -t before reloading. The match is intentionally simple and can match a spoofed header. Nginx configuration patterns should be reviewed against your installed version and existing routing; avoid adding a broad rule to a shared reverse proxy without checking other virtual hosts.

When blocking is the right choice

Blocking GPTBot can be a reasonable policy if you do not want public pages included in the training use OpenAI describes. Consider what you give up before expanding the rule to other tokens:

• GPTBot: OpenAI’s training crawler. A specific block does not itself block the separately controlled search crawler.
• OAI-SearchBot: used to surface sites in ChatGPT Search. OpenAI says opting out means your site will not appear in ChatGPT Search results, though it may still appear as a navigational link.
• ChatGPT-User: fetches pages for certain user actions. OpenAI says these requests are user initiated, not automatic crawling, and robots.txt may not apply to them.

These descriptions and controls are from OpenAI’s current bot overview. The equivalent distinction exists at other providers too; see the AI crawler user-agent list. “Block AI” switches can combine several roles, so inspect what the chosen product actually blocks.

Verify the result

1. Fetch https://your-domain.example/robots.txt and confirm the GPTBot group is present in the served file.
2. Check Cloudflare or origin logs for the intended response to a verified test request; do not infer enforcement from robots.txt alone.
3. Check that unrelated crawlers and ordinary customers still reach public product pages.
4. Revisit the rule if the provider changes tokens or publishes new IP ranges. Never treat a user-agent string by itself as reliable identity.

For Shopify, theme-managed robots.txt is the starting point; an edge rule may need platform-specific setup. See the Shopify AI crawler guide.

Questions

Keep reading