ShipworkSite inspection
Checks

Crawlability

PaginationAre paginated pages set up right?Internal searchIs site search eating crawl budget?URL parametersAre parameters creating duplicates?Log analyserWhat does Googlebot actually crawl?Robots builderNeed a robots.txt file?Soft 404sAre pages “not found” but returning 200?JavaScriptCan crawlers see it without running JS?FreshnessIs my site quietly going stale?Robots.txtDoes robots.txt say what I think?

Indexing

Sitemap lastmodAre my sitemap dates valid?SitemapIs my sitemap actually fetchable?IndexingIs Google allowed to index my pages?CanonicalsIs Google indexing the wrong URL?Index signalsDo sitemap and index signals agree?Sitemap generatorNeed a sitemap.xml for my site?Bulk URL statusWhere does each of these URLs really land?Redirect builderNeed the redirect rules?RedirectsAre my URLs answering directly, over HTTPS?

On-page

Keyword ideasWhat are people searching for?SERP previewHow does my page look in Google?Content qualityAre your pages too thin or too alike?Image weightAre images slowing the page?CannibalizationAre pages competing with each other?On-page checkDoes the page use its target phrase?ImagesAre my images accessible and loading?Meta tag builderWhat should my title and share tags say?DuplicatesDo my pages compete for one query?

Links

Orphan pagesWhich pages can no link reach?Link graphHow deep do your pages sit?Anchor textDo links say what they point at?Outbound linksAre external links still alive?Broken linksAre internal links sending visitors nowhere?

Structured data

Rich resultsIs my markup eligible for a rich result?Structured dataIs my product schema valid?Schema coverageDo my key pages carry structured data?Social previewHow does my page look when shared?Schema vs pageDoes markup match the page price?Schema builderNeed valid JSON-LD?

International

Hreflang sitemapDo page and sitemap hreflang agree?HreflangDo my language versions link back?Hreflang builderNeed the hreflang tags?Page DoctorWhy is this page not doing well?Redirect planMy site has dead links — what redirects do I write?Sitemap diffIs anything missing from my sitemap?Robots simulatorWhat does my robots.txt actually block?Can Googlebot?Can Googlebot fetch this URL?
Every error, explainedGuides
Pricing Learn Error guides Run free audit

+1 free check

Welcome to Shipwork

Sign in free and your next check this month is on us.

Continue with GoogleContinue with email

We email you a 6-digit code. No password.

No card needed. Reports and watches follow you to any device.

ChatGPT-User: block or allow OpenAI’s user-triggered fetch agent

ChatGPT-User is a user-agent token operated by OpenAI. Fetches a page when a ChatGPT user or custom GPT asks for it, rather than crawling automatically.

Reads your public robots.txt. Nothing is stored.

What ChatGPT-User does

Fetches a page when a ChatGPT user or custom GPT asks for it, rather than crawling automatically.

Because a person triggers these fetches, OpenAI documents that robots.txt rules may not apply to them in the same way. Check its current crawler documentation.

How to block ChatGPT-User in robots.txt

Add these two lines to the robots.txt file at the root of your domain: User-agent: ChatGPT-User and, on the next line, Disallow: /

To block only part of the site, replace / with a path such as /private/. Rules apply per host, so a subdomain needs its own robots.txt.

How to allow ChatGPT-User

If you have no rule for ChatGPT-User and no blanket User-agent: * Disallow: /, it is already allowed. To allow it explicitly while blocking other bots, add User-agent: ChatGPT-User followed by Allow: /

Check it worked

  1. Publish the updated robots.txt and open it in a browser at /robots.txt to confirm the live file has the rule.
  2. Run the free AI crawler check: it reads your live robots.txt and reports whether ChatGPT-User is allowed or blocked.
  3. Look for ChatGPT-User in your server or CDN logs over the following weeks to see whether requests stop.

robots.txt is a public request, not access control. OpenAI publishes its own documentation for ChatGPT-User; confirm the current token and behaviour there, since operators change them.

Related explainers

Questions

Who operates ChatGPT-User?
OpenAI. Fetches a page when a ChatGPT user or custom GPT asks for it, rather than crawling automatically.
Does blocking ChatGPT-User affect Google rankings?
No. ChatGPT-User is not Googlebot, so blocking it has no effect on Google Search.
How do I check if ChatGPT-User is blocked on my site?
Run the free AI crawler check on your domain. It reads the live robots.txt and reports the result for ChatGPT-User and the other major AI agents.