Generate your robots.txt

Set the open paths, the closed paths and an AI crawler stance, and get one complete file. Copy it to your domain root, then validate it live.

After generating

Serve it at /robots.txt on the exact host, then run the robots check. It resolves every rule per URL the way each crawler does and names what is actually blocked, including the AI agents.

What the robots.txt generator includes

The form builds one robots.txt file from four inputs: a sitemap URL, paths to disallow, paths to explicitly allow, and a preset AI-crawler stance. The default path examples close areas such as /admin/, /cart/, /checkout/, /account/ and /search/, while explicitly allowing common product, collection, page and blog paths. You can replace those examples with paths that match your site.

The AI choice creates named groups for GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended, OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot. The preset can allow all, block all, or allow selected search/fetch agents while blocking the listed training agents. The output is plain text to copy or download; it does not publish the file or inspect your existing rules.

How to read the generated rules

A robots.txt group starts with User-agent:, followed by directives that apply to the matching crawler. Disallow: /private/ asks compliant crawlers not to fetch URLs under that path; Allow: /products/ explicitly keeps that path open. A Sitemap: line gives crawlers the sitemap URL. The generator puts general Allow and Disallow lines under User-agent: *, then adds named groups for its AI preset.

Rules are crawl instructions, not access protection. Anyone can still request a disallowed URL directly, and robots.txt does not remove a URL already known to a search engine. Never use it to protect account pages, private documents or customer data; use authentication and server permissions. Rule matching has details, so test the actual published file with the robots check rather than assuming the generated text has the intended effect.

Choose a crawler stance deliberately

The open-search preset is not the same as allowing every AI crawler. It blocks the generator’s listed training-oriented agents and allows selected search or user-requested agents. That can preserve access by search crawlers while expressing a separate preference for training crawlers. Providers change tokens and purposes over time, so check their current documentation before relying on a preset as a complete inventory.

Blocking all listed AI agents can remove access to search-time crawlers as well as training crawlers. Before selecting it, decide whether ChatGPT Search, Perplexity search or Claude search visibility matters to you. The generator has fixed presets; it does not offer a per-agent editor or explain every vendor’s policy. For a nuanced decision, review AI crawler access and the current AI crawler user-agent reference.

Common robots.txt mistakes and fixes

A copied Disallow: / under User-agent: * can prevent compliant crawlers from fetching the whole site. Remove it or narrow the path if you meant to close only a section. Another frequent error is adding paths with no leading slash or copying example paths without checking whether they exist. A sitemap URL should be an absolute public URL, such as https://example.com/sitemap.xml.

If the file already contains hand-written groups, replacing it wholesale may discard important rules or app-generated directives. Compare old and new output before deployment. Some platforms manage robots.txt through a theme template or an admin setting; use the platform’s supported editing method and verify what the root URL serves afterward. Avoid repeated conflicting groups: explicit rules can be interpreted by specificity and matching length, so an apparently broad Allow does not necessarily override a more specific block.

Publish and verify the file

Save the output as plain text at the root of the exact hostname you want to control, for example https://example.com/robots.txt. A www subdomain and an apex domain can serve different files, so test the canonical host and any alternate hostname separately. Confirm the response is public text and not a theme preview, login page, redirect loop or cached old version.

Then run the robots check to see what the live file says and use robots test for specific URL decisions. If Google indexing is the concern, check Search Console URL Inspection too: robots.txt controls crawling and is not a noindex directive. The robots.txt vs. noindex guide explains the distinction.

Questions

What does robots.txt actually control?

Which paths crawlers may fetch. It is a request, not enforcement. Well-behaved crawlers obey it, malicious ones ignore it, so it is access management, not security.

How do I handle AI crawlers?

Decide per agent, not in one sweep. Search-time agents that cite sources are worth allowing; training crawlers are a separate call. This generator writes a named group per agent so each gets its own verdict.

Will blocking a bot hurt my SEO?

Blocking Googlebot hurts search. Blocking a training crawler does not touch rankings. They are separate user-agents with separate consequences, which is why the generator keeps them in separate groups.

How do I test the file?

Paste it live, then run the robots check: it resolves your rules per URL the way each crawler does and names what is actually blocked.

How do I create a robots.txt file for my website?

List paths that crawlers should not fetch, paths that must stay available, and the public sitemap URL. Choose an AI stance only after deciding which named agents you want to allow. Generate the file, publish it as plain text at the root of each relevant hostname, then fetch /robots.txt directly to confirm the deployed contents.

Should I block AI crawlers in robots.txt?

That depends on the crawler and your goals. A training crawler and a search crawler can be separate user agents, so a blanket block may remove search visibility you wanted to keep. The generator offers broad presets rather than custom per-agent choices. Check each provider's current bot documentation and test the published rules before deciding.

Does robots.txt protect a private page?

No. It asks compliant crawlers not to fetch a URL, but the file is public and does not restrict a direct request. Protect private content with authentication or server-side access controls. Do not put confidential paths or data in robots.txt as a security measure.

Why is my sitemap missing from the generated file?

The generator adds a Sitemap line only when the sitemap field is non-empty. Enter the full public URL, including https://, then generate again. After publishing, fetch the live robots.txt and confirm the line survived your platform's template or deployment process.

Can I replace my existing robots.txt with this output?

Do not replace it without reviewing the current file. Existing rules may exclude private application paths from crawl, support a sitemap, or have platform-specific requirements. Compare each old directive with the generated result, preserve necessary rules, publish through the supported platform method, and validate the live version.