BYTETOOLS

Robots.txt Generator

Build a robots.txt file with user-agent rules, allow/disallow paths, crawl-delay and sitemap URL. One-click presets to allow all, block all, or block AI bots.

Rule 1

robots.txt

What is the Robots.txt Generator?

The Robots. txt Generator lets you compose a valid robots.

  • Multiple user-agent rule groups with allow/disallow paths
  • Presets: Allow all, Block all, Block AI bots
  • Crawl-delay and Sitemap directives supported
  • Live preview with copy and download
  • Covers GPTBot, CCBot, anthropic-ai, Google-Extended and more
  • Runs fully client-side

How to use the Robots.txt Generator

  1. 1

    Start from a preset (Allow all, Block all, Block AI bots) or build your own rules.

  2. 2

    For each rule group set the user-agent and list the paths to allow or disallow.

  3. 3

    Optionally add a crawl-delay and your sitemap URL.

  4. 4

    Copy the generated file or download robots.txt.

  5. 5

    Upload it to the root of your domain, e.g. https://example.com/robots.txt.

About the Robots.txt Generator

The Robots.txt Generator lets you compose a valid robots.txt file from simple rule groups: pick a user-agent, add allow and disallow paths, set an optional crawl-delay and sitemap URL, and the file output updates live.

One-click presets cover the common cases — allow everything, block everything, or block the major AI training crawlers (GPTBot, CCBot, anthropic-ai, Google-Extended and friends) while leaving normal search bots untouched.

The generator runs entirely in your browser and the result downloads as robots.txt, ready to upload to your site root. It suits site owners who want crawl control without memorising the robots exclusion protocol.

Frequently asked questions

Does robots.txt stop a page from appearing in Google?

Not reliably. Robots.txt blocks crawling, not indexing — a blocked URL can still appear in results (without a snippet) if other sites link to it. To keep a page out of search, allow crawling and use a noindex meta tag, or protect it with authentication.

How do I block AI bots from training on my content?

Add disallow rules for the AI crawlers' user-agents: GPTBot (OpenAI), CCBot (Common Crawl), anthropic-ai and ClaudeBot (Anthropic), Google-Extended (Gemini training), PerplexityBot and Bytespider. Our 'Block AI bots' preset adds these in one click. Compliance is voluntary, but reputable companies honour it.

Where does the robots.txt file go?

At the root of the host: https://example.com/robots.txt. It must be exactly that path and filename — a robots.txt in a subdirectory is ignored. Each subdomain (blog.example.com) needs its own file.

Does Google support crawl-delay?

No — Googlebot ignores crawl-delay; you control Google's crawl rate via Search Console instead. Bing and Yandex do respect it. It's safe to include for those crawlers, but don't expect it to slow Googlebot down.

What does 'Disallow:' with an empty value mean?

An empty Disallow means 'nothing is disallowed' — i.e. allow everything for that user-agent. Conversely, 'Disallow: /' blocks the entire site. The single slash is the difference between fully open and fully closed, so double-check it before publishing.

Should I list my sitemap in robots.txt?

Yes, it's an easy win. The 'Sitemap:' directive (with an absolute URL) helps all search engines discover your sitemap without you submitting it to each webmaster console, and multiple Sitemap lines are allowed.

Guides for Robots.txt Generator

Related tools