Custom tools
Praha, CZ
AI crawlers · Seznam

robots.txt generator.

The generator creates a robots.txt file that tells search engines and AI crawlers where they may go on your site. It has three presets: allow everyone, allow AI search but block model training, or block all AI. It covers GPTBot, ClaudeBot, PerplexityBot, Google‑Extended and more, and instantly tests whether a chosen path is allowed.

Starting point
AI crawlers
Only block technical paths. robots.txt is not security; protect private pages with a password.
robots.txt

—

Runs entirely in your browser. Nothing is uploaded or stored.

How to use it

04
  1. Step01

    Pick a preset

    For most businesses we recommend allowing everyone: visibility in AI answers is an advantage today.

  2. Step02

    Add blocked paths

    Technical ones only: admin, cart, API. Not pages you want in search.

  3. Step03

    Add your sitemap

    The full URL, for example https://www.yoursite.com/sitemap.xml.

  4. Step04

    Test and upload

    The file belongs at the root: https://www.yoursite.com/robots.txt.

Who’s who among crawlers

CrawlerCompanyPurpose
Googlebot, Bingbot, SeznamBotGoogle, Microsoft, SeznamClassic search
OAI-SearchBot, Claude-SearchBot, PerplexityBotOpenAI, Anthropic, PerplexityAI search answers
ChatGPT-User, Claude-UserOpenAI, AnthropicFetching a page on a user’s request
GPTBot, ClaudeBot, Google-Extended, CCBotOpenAI, Anthropic, Google, Common CrawlTraining language models

How the generator decides

It uses the same rules as search engines under RFC 9309: the longest matching path wins, Allow wins ties, an asterisk matches any text and $ marks the end of the URL. The tester on the right shows whether the selected crawler may fetch a given path.

How is your current site doing? Check it with the AI visibility checker, which loads and evaluates robots.txt for you.

FAQ05

Questions people ask

01Should I block AI crawlers?

It depends on what you want. If you block them, AI assistants like ChatGPT or Perplexity won't cite or recommend you. For most businesses that want customers, it's better to allow them. If only model training bothers you, use the "Search yes, AI training no" preset.

02What's the difference between GPTBot and OAI-SearchBot?

GPTBot collects data for training OpenAI models. OAI-SearchBot indexes the web for search in ChatGPT. ChatGPT-User fetches a page when a user asks about it in chat. Anthropic similarly separates ClaudeBot, Claude-SearchBot and Claude-User.

03Does robots.txt block access to a page?

No. It's a request that polite robots respect. For real protection use a password. A blocked page can also appear in results without a description if other pages link to it. To remove a page from the index, use meta robots noindex.

04What is Google-Extended?

A token Google uses to separate use of your content for Gemini models. Blocking it doesn't affect your rankings in Google Search.

05Does Seznam need special rules?

No, SeznamBot follows the standard rules. Just don't block it by accident; for Czech websites Seznam is still an important source of visits.

Next step01

What next

Custom
01.

A custom tool

Need something that is not here? I build custom tools. Contact is being set up right now.
Related06

Keep going

Got work a program should be doing?

I am still setting up contact. Meanwhile, try the tools that already work.

Contact

Free tools