Crawled Currently Not Indexed — Field Notes for WordPress Teams

Crawled Currently Not Indexed is one of the small levers that pays back at scale. Here's the version we run in client work, tuned for AI-search visibility without breaking existing rankings.

Context First

Crawled Currently Not Indexed matters more today than it did two years ago because AI-search rewards the underlying structure this work produces.

The Playbook

Four steps, in order.

• Detect the exact pages affected.
• Explain the finding in plain English.
• Ship a reversible fix behind an Approve gate.
• Track the metric that moves.

Pitfalls to Avoid

Most damage comes from irreversible, un-audited changes. Every step above is bounded so a rollback is a click, not a project.

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://wbpomniseo.com/sitemap.xml

robots.txt — allow AI crawlers you want citing you

• Install WBP Omni SEO Pro on staging and run the scanner against one silo.
• Approve the first 10 low-risk fixes (missing alt text, canonical, breadcrumbs, schema).
• Roll one fix back on purpose to feel the safety net before you scale.
• Verify with Bot Tracker that GPTBot, ClaudeBot and PerplexityBot have re-crawled the fixed URLs.
• Promote the workflow to production and schedule the weekly per-silo run.
• Log-file check: confirm the fix reached Googlebot AND the AI crawlers, not just the frontend.

Inside WBP Omni SEO Pro: Performance — Health, Conflicts & Migration

Why this matters for "Crawled Currently Not Indexed — Field Notes for WordPress Teams": Two SEO plugins active at once is the leading cause of duplicate canonicals, conflicting schema and missing sitemaps.

Insights & analysis

Teams pulling ahead in AI search share three habits: they treat schema as a contract, they treat internal links as a graph problem, and they treat every applied fix as reversible. Everything else — tools, dashboards, agencies — is downstream of those three.

Stats snapshot

References & further reading

• Google Search Central — Structured data guidelines
• web.dev — Core Web Vitals field data
• Search Engine Journal — AI Overviews coverage
• Wikipedia — Semantic search, entity linking, schema.org
• YouTube: WP Bulk Publishing channel — walkthroughs of the agentic loop
• Reddit — r/SEO, r/bigseo threads on GEO measurement

Paired module: Cloudflare Edge Integration

Push redirects, security rules and cache rules to the Cloudflare edge, and read edge analytics back into WBP. Redirects and rules at the edge are 10× faster than at the origin and remove the origin as a bottleneck for SEO plumbing.

• Integrations → Connect Cloudflare
• Choose which rules push to edge (redirects, security, cache)
• Watch edge hit rate and rule performance
• Rollback pushes zone-side without touching Cloudflare UI

Real-world examples

Three shapes this problem takes in the wild — and what the fix looked like when a team applied the Technical SEO playbook end-to-end.

The workflow at a glance

Final thoughts

Treat Crawled Currently Not Indexed as infrastructure, not a checklist. When the loop is automated and every change has an owner, both Google and the LLMs stop treating your site as noise and start treating it as a source.