Crawl Budget Tuning — In Production

Crawl Budget Tuning is one of the small levers that pays back at scale. Here's the version we run in client work, tuned for AI-search visibility without breaking existing rankings.

Why This Keeps Coming Back

Crawl Budget Tuning shows up in every WordPress audit because teams treat it as a launch task. It isn't — it's a recurring loop.

The Loop That Works

Detect → Explain → Fix → Approve → Apply → Track → Rollback. The loop is the product, not the report.

What to Stop Doing

Stop shipping PDF audits nobody reads. Ship diffs a human can approve.

# nginx.conf — rewrite legacy URLs at the edge
location ~ ^/old/(.+)$ {
  return 301 https://$host/new/$1;
}

Edge-evaluated 301 rewrite — faster than PHP origin

• Install WBP Omni SEO Pro on staging and run the scanner against one silo.
• Approve the first 10 low-risk fixes (missing alt text, canonical, breadcrumbs, schema).
• Roll one fix back on purpose to feel the safety net before you scale.
• Verify with Bot Tracker that GPTBot, ClaudeBot and PerplexityBot have re-crawled the fixed URLs.
• Promote the workflow to production and schedule the weekly per-silo run.
• Log-file check: confirm the fix reached Googlebot AND the AI crawlers, not just the frontend.

Inside WBP Omni SEO Pro: GEO — Generative Engine Optimisation

Why this matters for "Crawl Budget Tuning — In Production": AI answer engines route intent before the SERP does; being invisible to them is being invisible to the top of the funnel.

References & further reading

• Google Search Central — Structured data guidelines
• web.dev — Core Web Vitals field data
• Search Engine Journal — AI Overviews coverage
• Wikipedia — Semantic search, entity linking, schema.org
• YouTube: WP Bulk Publishing channel — walkthroughs of the agentic loop
• Reddit — r/SEO, r/bigseo threads on GEO measurement

Where this is heading (2026 → 2027)

1. Citation-attribution becomes a first-class metric alongside clicks.
2. Schema graphs consolidate — one @graph per URL, enforced by search engines.
3. Reversible, human-in-the-loop agents become the compliance default.
4. Programmatic pages without unique data get filtered pre-index.

Paired module: Brand Authority (Fact Bank + Entities)

A structured store of your brand facts, statistics, quotes, entity graph and NAP that feeds schema, AI answers and content briefs. LLMs cite sources they can reconcile; a Fact Bank makes your facts reconcilable across every page.

• Brand Authority → Fact Bank → Add facts with citations
• Attach entities and sameAs targets to key concepts
• Reference facts in posts with a shortcode or block
• Expose the Fact Bank as machine-readable JSON for LLMs

Real-world examples

Three shapes this problem takes in the wild — and what the fix looked like when a team applied the Technical SEO playbook end-to-end.

The workflow at a glance

Final thoughts

Treat Crawl Budget Tuning as infrastructure, not a checklist. When the loop is automated and every change has an owner, both Google and the LLMs stop treating your site as noise and start treating it as a source.