What does robots.txt actually control?

Which paths crawlers may fetch. It is a request, not enforcement. Well-behaved crawlers obey it, malicious ones ignore it, so it is access management, not security.

How do I handle AI crawlers?

Decide per agent, not in one sweep. Search-time agents that cite sources are worth allowing; training crawlers are a separate call. This generator writes a named group per agent so each gets its own verdict.

Will blocking a bot hurt my SEO?

Blocking Googlebot hurts search. Blocking a training crawler does not touch rankings. They are separate user-agents with separate consequences, which is why the generator keeps them in separate groups.

How do I test the file?

Paste it live, then run the robots check: it resolves your rules per URL the way each crawler does and names what is actually blocked.

Generate your robots.txt

Three inputs, one file: which paths stay open, which stay closed, and what each AI crawler gets. Copy it to your domain root, then validate it live.

After generating

Serve it at /robots.txt on the exact host, then run the robots check. It resolves every rule per URL the way each crawler does and names what is actually blocked, including the AI agents.

Questions

What does robots.txt actually control?: Which paths crawlers may fetch. It is a request, not enforcement. Well-behaved crawlers obey it, malicious ones ignore it, so it is access management, not security.
How do I handle AI crawlers?: Decide per agent, not in one sweep. Search-time agents that cite sources are worth allowing; training crawlers are a separate call. This generator writes a named group per agent so each gets its own verdict.
Will blocking a bot hurt my SEO?: Blocking Googlebot hurts search. Blocking a training crawler does not touch rankings. They are separate user-agents with separate consequences, which is why the generator keeps them in separate groups.
How do I test the file?: Paste it live, then run the robots check: it resolves your rules per URL the way each crawler does and names what is actually blocked.