# Robots.txt Generator (with AI Crawlers)

> Build a clean robots.txt with your sitemap, disallowed paths and per-bot controls for OpenAI, Anthropic, Perplexity, Google and Common Crawl.

```yaml
url: "https://ailaunch.dev/tools/robots-txt-generator"
slug: robots-txt-generator
category: geo
keywords: "robots.txt generator, block gptbot, allow claudebot, ai crawler robots.txt, robots txt ai"
```

## How to use

1. **Add your sitemap** - Full URL to sitemap.xml.
2. **Disallow private paths** - Admin, API, checkout; one per line.
3. **Pick AI crawler rules** - Allow AI search bots, optionally block training-only bots.

## Training bots vs search bots

AI companies run separate crawlers for training and for live answers. OpenAI uses GPTBot for training and OAI-SearchBot / ChatGPT-User for search and browsing; Anthropic has ClaudeBot and Claude-SearchBot; Google uses Google-Extended as the opt-out token for Gemini training. Blocking only the training bots keeps you citable in AI answers.

## What you get

- Presets: allow all, block training only, block all AI
- 9 AI user agents
- Sitemap + disallow paths
- Copy or download robots.txt

## FAQ

### Will blocking AI bots hurt Google rankings?

No. Google-Extended only controls Gemini training use; Googlebot (search) is unaffected.

### Do all bots obey robots.txt?

Major AI companies state they honor it. Unknown scrapers may not; use your CDN's bot rules for those.

## Related tools

- **[ai.txt Generator](https://ailaunch.dev/tools/ai-txt-generator)** - Create an ai.txt file stating how AI systems may use your content: training, summarizing and citing, with attribution preferences.
- **[AI Search Readiness Checker](https://ailaunch.dev/tools/ai-search-readiness-checker)** - Check whether AI crawlers can access your site and whether it has llms.txt, ai.txt, structured data and the metadata LLMs use to cite you.
- **[XML Sitemap Generator](https://ailaunch.dev/tools/sitemap-generator)** - Paste a list of URLs and get a standards-compliant sitemap.xml with optional changefreq, priority and lastmod.