AI Crawler Robots.txt Generator
One robots.txt with rules for 20+ AI crawlers.
✅ 100% free✅ No signup✅ Runs in your browser✅ No data sent to servers
Output
User-agent: GPTBot # Allow or Disallow — decide per bot Allow: / User-agent: ChatGPT-User # Allow or Disallow — decide per bot Allow: / User-agent: OAI-SearchBot # Allow or Disallow — decide per bot Allow: / User-agent: ClaudeBot # Allow or Disallow — decide per bot Allow: / User-agent: Claude-Web # Allow or Disallow — decide per bot Allow: / User-agent: PerplexityBot # Allow or Disallow — decide per bot Allow: / User-agent: Perplexity-User # Allow or Disallow — decide per bot Allow: / User-agent: Google-Extended # Allow or Disallow — decide per bot Allow: / User-agent: Applebot-Extended # Allow or Disallow — decide per bot Allow: / User-agent: CCBot # Allow or Disallow — decide per bot Allow: / User-agent: Bytespider # Allow or Disallow — decide per bot Allow: / User-agent: Meta-ExternalAgent # Allow or Disallow — decide per bot Allow: / User-agent: Amazonbot # Allow or Disallow — decide per bot Allow: / User-agent: DuckAssistBot # Allow or Disallow — decide per bot Allow: / User-agent: cohere-ai # Allow or Disallow — decide per bot Allow: / User-agent: YouBot # Allow or Disallow — decide per bot Allow: / User-agent: Diffbot # Allow or Disallow — decide per bot Allow: / User-agent: Omgilibot # Allow or Disallow — decide per bot Allow: / User-agent: FacebookBot # Allow or Disallow — decide per bot Allow: / User-agent: Timpibot # Allow or Disallow — decide per bot Allow: /
Why this tool survives the ChatGPT era
Every site needs an explicit stance on AI training + AI-answer crawlers. This is that stance.
Frequently asked
Block or allow?
Allow OAI-SearchBot / PerplexityBot for citations; block GPTBot / CCBot for training. Your call.
Ship all of this at scale with WBP Omni SEO Pro
Every generator on this page can auto-apply across thousands of WordPress URLs — schema, meta, hreflang, robots, llms.txt — with GEO, AEO and AIO built in.
See pricing →Related AI Search tools
l
llms.txt Generator
Generate an llms.txt manifest so LLM crawlers ingest your best pages first.
A
AI Answer Formatter
Rewrite paragraphs into Q&A blocks LLMs love to cite.
C
Citation Formatter
Convert a URL + title into ChatGPT-, Perplexity- and Claude-friendly citations.
P
Prompt SEO Optimizer
Turn a target keyword into a set of prompts that trigger AI Overviews.
G
Google Business Profile Optimizer
Rewrite your GBP description to 750 chars, keyword-safe, spam-filter-safe.
l
llms.txt Audit
Manual audit checklist for /llms.txt — the emerging AI-crawler standard.
Where teams ship this in production
Real sites using ai crawler robots.txt generator-style patterns for AEO/GEO visibility.
shopify.com