AI Bot Access Analyzer

Check whether AI crawlers from OpenAI, Anthropic, Perplexity, Google (Gemini), Microsoft, and Meta can actually reach your website pages.

This is based on what your server and CDN actually return to each bot, not just what robots.txt says.

The reality of AI bot access

Your robots.txt might say “welcome,” but CDNs, firewalls, and bot-management rules often quietly block AI crawlers at the infrastructure level — so you lose AI visibility without ever knowing.

What we test

We send each bot’s real user-agent to your URL and reconcile the live HTTP response with robots.txt, robots meta tags, and X-Robots-Tag headers — across the major AI vendors.

How to read results

allowed Bot can crawl & reference your content. blocked Stopped by HTTP, robots.txt, or a directive. error Connection failed — often bot-specific rejection. directive robots.txt opt-out only (no live crawler).

Known AI crawlers

OpenAI

GPTBottrain
Crawls content to train OpenAI models.
If blocked: Content excluded from GPT/ChatGPT training data.
ChatGPT-Userfetch
Live fetch when a ChatGPT user’s prompt visits a page.
If blocked: ChatGPT can’t read your page in real time.
OAI-SearchBotsearch
Indexes pages for ChatGPT Search.
If blocked: Excluded from ChatGPT Search results/citations.

Anthropic

ClaudeBottrain
Crawls content to train Anthropic’s Claude models.
If blocked: Content not used to improve Claude.
Claude-Userfetch
Live fetch when a Claude user references a page.
If blocked: Claude can’t browse your content for users.
Claude-SearchBotsearch
Indexes pages for Claude’s search features.
If blocked: Excluded from Claude search citations.

Perplexity

PerplexityBotsearch
Indexes pages so they can surface as Perplexity citations.
If blocked: No inclusion in Perplexity answers.
Perplexity-Userfetch
Live fetch when a Perplexity user’s action visits a page.
If blocked: No real-time citations in Perplexity answers.

Google / Gemini

Googlebotsearch
Google Search index — also powers AI Overviews & Gemini grounding.
If blocked: Dropped from Google Search and AI Overviews.
Google-Extendedtrain
robots.txt control for Gemini & Vertex AI training use.
If blocked: Content not used to train Gemini / Vertex AI.
Google-CloudVertexBotsearch
Fetches pages for Vertex AI Search / Gemini grounding apps.
If blocked: Unavailable to Vertex AI Search & grounded Gemini apps.

Microsoft

Bingbotsearch
Bing index — powers Microsoft Copilot answers.
If blocked: Dropped from Bing and Copilot answers.

Meta

meta-externalagenttrain
Crawls content for Meta AI and Llama models.
If blocked: Content not used by Meta AI / Llama.
Limitations. This test sends each bot’s real user-agent string but from the tool’s server IP — genuine crawlers come from published, verified IP ranges, so a site that filters by verified IP may behave differently for the real bot. Results can also vary by geography and CDN caching. Treat results as directional, not definitive.