# robots.txt Library by CMS
Maintained by WBP Omni SEO — https://wbpomniseo.com/templates/robots-txt-library
License: CC BY 4.0.

---

## WordPress (recommended default)
```
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /wp-json/
Disallow: /trackback/
Disallow: /*?replytocom=
Disallow: /author/
Disallow: /tag/*/page/
Disallow: /category/*/page/

Sitemap: https://example.com/sitemap.xml
```

## Shopify
```
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkout
Disallow: /search
Disallow: /*?*variant=
Disallow: /*?*ref=
Disallow: /policies/

Sitemap: https://example.com/sitemap.xml
```

## Webflow
```
User-agent: *
Disallow: /?
Disallow: /*?
Allow: /

Sitemap: https://example.com/sitemap.xml
```

## Wix
```
User-agent: *
Disallow: /_api/
Disallow: /_functions*
Disallow: /_partials*

Sitemap: https://example.com/sitemap.xml
```

## Squarespace
```
User-agent: *
Disallow: /config
Disallow: /commerce
Disallow: /search

Sitemap: https://example.com/sitemap.xml
```

## Ghost
```
User-agent: *
Disallow: /ghost/
Disallow: /p/

Sitemap: https://example.com/sitemap.xml
```

## Framer
```
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
```

## Next.js / TanStack Start (SPA)
```
User-agent: *
Allow: /
Disallow: /api/
Disallow: /_next/

Sitemap: https://example.com/sitemap.xml
```

## AI crawler control (block AI training, allow AI answer citation)
```
# Block training crawlers
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /

# Allow AI answer engines (they need to fetch to cite you)
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
```

## AI-friendly (allow everything — best for citation SEO)
```
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
```

WBP Omni SEO Pro ships a visual robots.txt editor with per-bot rules and live GSC validation. https://wbpomniseo.com/pricing