Sign inLaunch

AI visibility (GEO) · Free · No signup

Robots.txt Generator (with AI Crawlers)

Build a clean robots.txt with your sitemap, disallowed paths and per-bot controls for OpenAI, Anthropic, Perplexity, Google and Common Crawl.

Ask AI
ChatGPTClaudePerplexityGrok

One per line.

AI crawlers

robots.txt

How to use

  1. Step 1

    Add your sitemap

    Full URL to sitemap.xml.

  2. Step 2

    Disallow private paths

    Admin, API, checkout; one per line.

  3. Step 3

    Pick AI crawler rules

    Allow AI search bots, optionally block training-only bots.

Training bots vs search bots

AI companies run separate crawlers for training and for live answers. OpenAI uses GPTBot for training and OAI-SearchBot / ChatGPT-User for search and browsing; Anthropic has ClaudeBot and Claude-SearchBot; Google uses Google-Extended as the opt-out token for Gemini training. Blocking only the training bots keeps you citable in AI answers.

What you get

  • ✓ Presets: allow all, block training only, block all AI
  • ✓ 9 AI user agents
  • ✓ Sitemap + disallow paths
  • ✓ Copy or download robots.txt

FAQ

Will blocking AI bots hurt Google rankings?

No. Google-Extended only controls Gemini training use; Googlebot (search) is unaffected.

Do all bots obey robots.txt?

Major AI companies state they honor it. Unknown scrapers may not; use your CDN's bot rules for those.

Related tools

ai.txt Generator

Create an ai.txt file stating how AI systems may use your content: training, summarizing and citing, with attribution preferences.

AI Search Readiness Checker

Check whether AI crawlers can access your site and whether it has llms.txt, ai.txt, structured data and the metadata LLMs use to cite you.

XML Sitemap Generator

Paste a list of URLs and get a standards-compliant sitemap.xml with optional changefreq, priority and lastmod.

Launching an AI product?

Get a permanent product page, a quality backlink, and discovery across Google and AI search. Free.

Launch on AI Launch

For agents: markdown version · llms.txt