Code library · Free · CC BY 4.0
robots.txt Library by CMS
Production-tested robots.txt for WordPress, Shopify, Webflow, Wix, Squarespace, Ghost, Framer, Next.js — plus AI-bot control
A misconfigured robots.txt is the fastest way to deindex a site. This library ships the exact rules that work in production for each major CMS, plus a separate AI-crawler section so you can block training bots without losing citations in ChatGPT and Perplexity answers.
What's inside
- 10 CMS-specific robots.txt files
- AI-crawler block-and-allow section (train vs cite)
- Notes on which sections to keep for GSC indexation
- Sitemap directive placement
Who it's for
- Site launches — ship a safe default
- Migrations — avoid accidental deindex
- Publishers weighing AI training vs AI citation
Ship it, don't just plan it.
WBP Omni SEO Pro ships a visual robots.txt editor with per-bot rules, GSC validation, and one-click AI-crawler control.
See WBP Omni SEO Pro pricingMore free templates
Google Sheet (CSV)
SEO Audit Template (Google Sheet)
A production-ready spreadsheet covering crawl, index, on-page, schema, Core Web Vitals, AI search, and E-E-A-T. Import into Google Sheets, Excel, or Notion.
Google Sheet (CSV)
Content Calendar Template
Twelve-week content calendar with keyword, intent, cluster, internal links, and CTA columns. Works in Google Sheets, Excel, Notion, or Airtable.
Google Sheet (CSV)
Programmatic SEO Keyword Sheet
The exact keyword sheet structure behind programmatic sites that rank. Two modifier columns, URL pattern, uniqueness signal, and priority.
Direct download: robots-txt-library.md