GPTBot is OpenAI's web crawler used to collect training data for GPT models and power real-time browsing in ChatGPT. User agent: GPTBot. It reads raw HTML without executing JavaScript. Allow via: User-agent: GPTBot / Allow: / in robots.txt.

What is PerplexityBot?

PerplexityBot is Perplexity AI's web crawler used to build and maintain the live search index that powers every Perplexity answer. User agent: PerplexityBot. It prioritizes recently updated content and crawls more aggressively than training-focused bots. Allow via: User-agent: PerplexityBot / Allow: /.

ClaudeBot is Anthropic's web crawler used to collect training data for Claude models. User agent: ClaudeBot. Like other AI crawlers, it reads raw HTML without JavaScript execution and respects robots.txt. Allow via: User-agent: ClaudeBot / Allow: /.

What is Google-Extended?

Google-Extended is Google's dedicated AI crawler, separate from Googlebot. It collects data for Gemini models and powers Google AI Overviews. Blocking Google-Extended does not affect Google Search rankings but removes eligibility for AI Overview citations. Allow via: User-agent: Google-Extended / Allow: /.

What is CCBot and why does it matter?

CCBot is Common Crawl's web crawler. Common Crawl maintains a public archive of billions of web pages that feeds training data for OpenAI, Anthropic, and many other AI companies simultaneously. Being indexed by CCBot increases your presence across multiple AI training datasets. Allow via: User-agent: CCBot / Allow: /.

How do I allow all AI crawlers at once?

Add individual User-agent stanzas for each crawler in robots.txt — GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Claude-Web, Google-Extended, Gemini-Deep-Research, CCBot, anthropic-ai, and Applebot-Extended. A blanket User-agent: * / Allow: / works but does not give you fine-grained control. Check for inherited Disallow: / rules from staging environments that may be blocking all crawlers.

AI Crawler Directory 2026 — Every AI Bot, User Agent, and robots.txt Rule

Complete AI crawler reference

AI crawlers fall into two categories: search crawlers that build real-time indexes for AI answer engines, and training crawlers that collect data for model training. Both affect your AI visibility in different ways.

Crawler	Company	User Agent	Type	Affects	robots.txt allow
GPTBot	OpenAI	`GPTBot`	Training	ChatGPT parametric memory, training data	`User-agent: GPTBot Allow: /`
OAI-SearchBot	OpenAI	`OAI-SearchBot`	Search	ChatGPT real-time browsing, SearchGPT	`User-agent: OAI-SearchBot Allow: /`
ChatGPT-User	OpenAI	`ChatGPT-User`	Search	ChatGPT web browsing plugin (user-triggered)	`User-agent: ChatGPT-User Allow: /`
PerplexityBot	Perplexity AI	`PerplexityBot`	Search	All Perplexity answers (live retrieval)	`User-agent: PerplexityBot Allow: /`
ClaudeBot	Anthropic	`ClaudeBot`	Training	Claude model training data	`User-agent: ClaudeBot Allow: /`
Claude-Web	Anthropic	`Claude-Web`	Search	Claude web search feature	`User-agent: Claude-Web Allow: /`
anthropic-ai	Anthropic	`anthropic-ai`	Training	Anthropic model training	`User-agent: anthropic-ai Allow: /`
Google-Extended	Google	`Google-Extended`	Both	Gemini training, Google AI Overviews	`User-agent: Google-Extended Allow: /`
Gemini-Deep-Research	Google	`Gemini-Deep-Research`	Search	Gemini Deep Research feature	`User-agent: Gemini-Deep-Research Allow: /`
Google-Agent	Google	`Google-Agent`	Search	Google AI agent products	`User-agent: Google-Agent Allow: /`
CCBot	Common Crawl	`CCBot`	Common Crawl	OpenAI, Anthropic, and many other AI training datasets simultaneously	`User-agent: CCBot Allow: /`
Applebot	Apple	`Applebot`	Search	Siri, Spotlight, Apple Intelligence search	`User-agent: Applebot Allow: /`
Applebot-Extended	Apple	`Applebot-Extended`	Training	Apple Intelligence model training	`User-agent: Applebot-Extended Allow: /`
PerplexityBot	Perplexity	`Perplexity-User`	Search	Perplexity user-triggered searches	`User-agent: Perplexity-User Allow: /`
YouBot	You.com	`YouBot`	Search	You.com AI search answers	`User-agent: YouBot Allow: /`
DuckAssistBot	DuckDuckGo	`DuckAssistBot`	Search	DuckDuckGo AI answers	`User-agent: DuckAssistBot Allow: /`
ExaBot	Exa.ai	`ExaBot`	Search	Exa AI search index	`User-agent: ExaBot Allow: /`
cohere-ai	Cohere	`cohere-ai`	Training	Cohere model training data	`User-agent: cohere-ai Allow: /`
DeepSeekBot	DeepSeek	`DeepSeekBot`	Training	DeepSeek model training data	`User-agent: DeepSeekBot Allow: /`

The complete robots.txt block — copy and paste

Add this to your existing robots.txt. If you already have a User-agent: * section, add these stanzas before it.

# ── OpenAI / ChatGPT User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # ── Perplexity User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # ── Anthropic / Claude User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / # ── Google AI User-agent: Google-Extended Allow: / User-agent: Gemini-Deep-Research Allow: / User-agent: Google-Agent Allow: / # ── Apple User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / # ── Common Crawl (feeds multiple AI training datasets) User-agent: CCBot Allow: / # ── Other AI search User-agent: YouBot Allow: / User-agent: DuckAssistBot Allow: / User-agent: ExaBot Allow: / User-agent: cohere-ai Allow: / User-agent: DeepSeekBot Allow: / # ── Sitemap Sitemap: https://brandioz.com/sitemap.xml

Search crawlers vs training crawlers — what's the difference?

Search crawlers (PerplexityBot, OAI-SearchBot, Claude-Web): Build real-time indexes used to answer user queries right now. Allowing these crawlers means your content can appear in AI answers within days of publishing. They re-crawl frequently, reward content freshness, and are directly responsible for live AI citations.
Training crawlers (GPTBot, ClaudeBot, anthropic-ai, CCBot): Collect data for model training runs. Content they index may not affect AI answers for months — until the next training run is complete. But training data presence is what makes parametric-first platforms (ChatGPT without browsing) able to describe your brand from memory.
CCBot (Common Crawl) — why it's uniquely important: Allowing CCBot is one allow rule that feeds training datasets for OpenAI, Anthropic, Cohere, AI2, and many other AI companies simultaneously. Common Crawl is the shared infrastructure of the AI training data ecosystem. A single CCBot Allow: / rule is the highest-leverage robots.txt change you can make for long-term parametric AI visibility.

How to verify crawlers are actually reaching your site

Check your server access logs for these user agent strings. If they don't appear, the crawlers are either blocked by robots.txt, hitting errors, or haven't discovered your site yet.

# Test if GPTBot can read your homepage curl -A "GPTBot" https://yourdomain.com | wc -w # Should return 600+. Under 600 = partial render problem. # Test if PerplexityBot can access your site curl -A "PerplexityBot" https://yourdomain.com -I # Should return HTTP 200. 403 means blocked. # Check your robots.txt is serving correctly curl https://yourdomain.com/robots.txt | grep -E "GPTBot|PerplexityBot|ClaudeBot" # Should return User-agent lines for each crawler

Frequently asked questions

Does blocking GPTBot affect ChatGPT answers about my brand?: Blocking GPTBot prevents OpenAI from crawling your site for training data and real-time browsing. It does not remove existing knowledge about your brand from ChatGPT's parametric memory — that comes from past training runs. But it prevents future updates to that knowledge and blocks real-time browsing citations.
Does blocking Google-Extended affect my Google Search rankings?: No. Google-Extended is separate from Googlebot. Blocking it has zero effect on Google Search rankings. It only affects eligibility for Google AI Overviews and Gemini model training.
Do AI crawlers respect robots.txt?: All major AI crawlers from named companies (OpenAI, Anthropic, Google, Perplexity, Apple) respect robots.txt. Scrapers and unofficial crawlers may not. The crawlers listed in this directory all have publicly documented user agent strings and policies.
How often do AI crawlers visit my site?: Search crawlers (PerplexityBot, OAI-SearchBot) re-crawl frequently — often within days of updates. Training crawlers (GPTBot, ClaudeBot, CCBot) crawl less predictably, driven by training data collection schedules. Pages discovered via sitemaps with recent lastmod dates tend to get crawled faster.