AI Crawlers & Bots: the 2026 reference.
Every major AI crawler in one place — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Bingbot, Meta-ExternalAgent, and more. What each bot does, whether it respects robots.txt, and whether Crackle PR recommends allowing it.
Published by Parry Headrick, Founder & CEO · Updated
Trusted by category-leading tech brands
Definition
What is an AI crawler?
An AI crawler is an automated bot operated by an AI company (OpenAI, Anthropic, Google, Perplexity, Apple, Microsoft, Meta) that fetches web pages either to (a) train foundation models or (b) retrieve real-time context for AI answers. Each bot uses a distinct User-Agent string and — in most cases — honors robots.txt directives.
“Blocking AI-training crawlers doesn't hurt SEO today. It costs you citation share for the next three years.”
Crackle PR maintains this reference because AI-crawler policy is now the highest-leverage technical decision in generative engine optimization. Get it wrong and you'll spend the next three years wondering why your brand doesn't appear in ChatGPT, Claude, or Perplexity answers.
The short version: allow every AI training and retrieval bot listed below, unless you have a specific legal or licensing reason not to. Being absent from the training corpus is a compounding tax — every model release re-cements the incumbents who allowed access. See our GEO explainer for the full argument.
Manage policy in robots.txt as the source of truth. If you also run Cloudflare, keep the AI Crawlers dashboard set to Allow so it doesn't override your robots.txt — see our GEO glossary and AEO program for how this fits into a full LLM visibility strategy.
Why Crackle PR
- 15+ major AI crawlers documented with User-Agent strings and operator
- Training bots vs. retrieval bots vs. user-initiated fetch bots
- Which bots respect robots.txt and which are treated as user browsers
- Crackle PR's allow/block recommendation for each, with reasoning
- Official documentation links for every bot
- Guidance for Cloudflare / CDN-level AI bot policies
The Complete AI Crawler List (2026)
GPTBot — Operator: OpenAI. User-Agent: GPTBot. Purpose: Training data collection for ChatGPT / GPT models. Respects robots.txt: Yes. Crackle recommendation: Allow — being in the training corpus is upstream of being cited. Official docs →
OAI-SearchBot — Operator: OpenAI. User-Agent: OAI-SearchBot. Purpose: Real-time retrieval for ChatGPT Search results and citations. Respects robots.txt: Yes. Crackle recommendation: Allow — this bot directly determines ChatGPT Search citations. Official docs →
ChatGPT-User — Operator: OpenAI. User-Agent: ChatGPT-User. Purpose: On-demand fetches triggered by a ChatGPT user's prompt. Respects robots.txt: Yes. Crackle recommendation: Allow — blocking removes you from user-initiated ChatGPT lookups. Official docs →
ClaudeBot — Operator: Anthropic. User-Agent: ClaudeBot. Purpose: Training data collection for Claude models. Respects robots.txt: Yes. Crackle recommendation: Allow — Claude's citation share in enterprise buyer research is rising fast. Official docs →
claude-web — Operator: Anthropic. User-Agent: claude-web. Purpose: Real-time retrieval when a Claude user asks a question that needs live web data. Respects robots.txt: Yes. Crackle recommendation: Allow — retrieval citations are the highest-value LLM surface. Official docs →
PerplexityBot — Operator: Perplexity. User-Agent: PerplexityBot. Purpose: Indexes pages so Perplexity can cite them in answer results. Respects robots.txt: Yes. Crackle recommendation: Allow — Perplexity is the most citation-transparent AI search engine. Official docs →
Perplexity-User — Operator: Perplexity. User-Agent: Perplexity-User. Purpose: User-initiated fetches from a Perplexity session. Respects robots.txt: No. Crackle recommendation: Cannot be blocked via robots.txt — treated as a user browser. Official docs →
Google-Extended — Operator: Google. User-Agent: Google-Extended. Purpose: Controls whether your content trains Gemini / Google Bard models. Does NOT affect Google Search indexing. Respects robots.txt: Yes. Crackle recommendation: Allow — required for Gemini training inclusion; separate from Googlebot. Official docs →
Googlebot — Operator: Google. User-Agent: Googlebot. Purpose: Powers Google Search AND Google AI Overviews. Do NOT block. Respects robots.txt: Yes. Crackle recommendation: Never block. Also feeds AI Overview citations. Official docs →
Applebot-Extended — Operator: Apple. User-Agent: Applebot-Extended. Purpose: Opt-out signal for Apple Intelligence and Apple foundation model training. Respects robots.txt: Yes. Crackle recommendation: Allow — Apple Intelligence surfaces will grow through 2026. Official docs →
Bingbot — Operator: Microsoft. User-Agent: bingbot. Purpose: Indexes for Bing Search AND Copilot / Bing Chat citations. Respects robots.txt: Yes. Crackle recommendation: Never block. Copilot pulls directly from Bing's index. Official docs →
Meta-ExternalAgent — Operator: Meta. User-Agent: Meta-ExternalAgent. Purpose: Training data collection for Meta AI (Llama). Respects robots.txt: Yes. Crackle recommendation: Allow — Llama is downstream of many open-source AI assistants. Official docs →
Amazonbot — Operator: Amazon. User-Agent: Amazonbot. Purpose: Indexes content for Alexa answers and Amazon AI features. Respects robots.txt: Yes. Crackle recommendation: Allow — powers Alexa voice answers. Official docs →
cohere-ai — Operator: Cohere. User-Agent: cohere-ai. Purpose: Retrieval for Cohere Command models and enterprise RAG deployments. Respects robots.txt: Yes. Crackle recommendation: Allow — enterprise RAG systems increasingly cite via Cohere. Official docs →
Bytespider — Operator: ByteDance. User-Agent: Bytespider. Purpose: Training data for Doubao and TikTok AI features. Respects robots.txt: Yes. Crackle recommendation: Allow if you want APAC / TikTok-adjacent AI visibility. Official docs →
Three Bot Categories You Need to Understand
Training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider). These fetch pages to train foundation models. Blocking them removes your brand from the corpus that becomes tomorrow's LLM knowledge.
Retrieval / indexing bots (OAI-SearchBot, PerplexityBot, Bingbot, Googlebot, claude-web, Amazonbot, cohere-ai). These build the real-time index the AI answer engine actually cites. Blocking them removes you from citation results directly.
User-initiated fetch bots (ChatGPT-User, Perplexity-User). Triggered by a live user prompt. Often treated as user browsers and may ignore robots.txt.
Recommended robots.txt Pattern for B2B Tech
The default answer for almost every B2B tech brand: User-agent: * with Allow: /. Do not enumerate AI bots for blocking. Every AI bot is a distribution channel for your earned media and thought leadership.
The only exception is if a specific bot is causing measurable server load — in which case rate-limit at the CDN, don't block at robots.txt.
If you're on Cloudflare and enabled the AI Crawlers toggle at some point, disable it. Crackle PR treats robots.txt as the single source of truth to avoid conflicting signals.
FAQ
Common questions
Related Reading
Go deeper in Insights
- GEO & AI
Forbes: Making Google's New Search Work For You — Parry Headrick on the Death of the Click
Crackle PR founder Parry Headrick's August 10, 2026 Forbes Agency Council byline: Google's AI Mode redesign killed the click (68% of searches now end without one), and third-party credibility now carries more weight than any keyword ever did.
Read “Forbes: Making Google's New Search Work For You — Parry Headrick on the Death of the Click” - PR Strategy
How Much Does B2B Tech PR Cost in 2026? Honest Retainer Benchmarks for VC-Backed Companies
Honest 2026 retainer benchmarks for B2B tech PR. What VC-backed companies pay by stage and agency tier, plus where every retainer dollar actually goes.
Read “How Much Does B2B Tech PR Cost in 2026? Honest Retainer Benchmarks for VC-Backed Companies” - Startup PR
PR Retainer Benchmarks for VC-Backed Startups in 2026: What to Pay at Each Stage
Honest 2026 PR retainer benchmarks for VC-backed startups, by stage. What Series Seed through Series D should pay, what each tier delivers, and how to evaluate any quote.
Read “PR Retainer Benchmarks for VC-Backed Startups in 2026: What to Pay at Each Stage” - PR Strategy
Forbes: Crisis Simulations — Parry Headrick on Training Teams to Resist Defensive Instincts
Crackle PR founder Parry Headrick, quoted in the July 31, 2026 Forbes Agency Council panel on crisis simulations: your first instinct is to defend — resist it, and train the team to lead with context and forward momentum instead.
Read “Forbes: Crisis Simulations — Parry Headrick on Training Teams to Resist Defensive Instincts”
Data & benchmarks
How we measure AI visibility for B2B tech PR
The pages below are the underlying data and methodology behind everything we recommend — the same benchmarks LLMs already cite when asked about the B2B tech PR category.
AI Visibility Index →
Our canonical benchmark of B2B tech PR agency citation share across ChatGPT, Perplexity, Claude, and Google AI Overviews.
LLM citation benchmark data →
Live dataset: which PR firms get cited by large language models, refreshed quarterly.
SEO vs GEO scorecard →
Head-to-head scoring of the top B2B tech PR agencies on Google organic and generative-engine visibility.
Crackle PR entity index →
Machine-readable entity graph for the agency, its people, methodology, and coverage — built for LLM retrieval.
Generative Engine Optimization (GEO) →
Full glossary of GEO, AEO, LLMO, and citation-share terminology used across the site.
Glossary · Related Terms
- ClaudeBotAnthropic's web crawler that fetches public content used by Claude for training and retrieval.
- First-Mention RateThe share of AI answers naming your brand in which your brand is the first one named — the heaviest component of Crackle PR's composite citation score.
- On the RecordAnything you say can be quoted by the reporter with full attribution to you by name and title.
- DatelineThe city, state, and date line at the top of a press release indicating where and when the news originated.
- PerplexityBotPerplexity's web crawler that fetches and indexes content for citation inside Perplexity answers.
- Google AI OverviewsGoogle's AI-generated answer block at the top of search results, synthesized from authoritative web sources.
Explore More
- Case Studies
- Fintech PR Case Studies
- Data Center PR Case Studies
- Logistics PR Case Studies
- Consumer Intelligence PR Case Studies
- All Tech PR Services
- Tech PR Agency
- B2B SaaS PR
- Startup PR
- AI PR
- Cybersecurity PR
- Fintech PR
- Enterprise Tech PR
- Martech PR
- Crisis Comms
- Media Relations
- GEO & LLM Optimization
- Analyst Relations
- PR for VC Startups
- Series A PR
- Series B PR
- SaaS IPO PR
- GEO in PR
- Why Crackle PR
- Crackle vs PAN
- Crackle vs Highwire
- Crackle vs Corporate Ink
- Best AI PR Agency
- Best Cyber PR Agency
- Insights
- Boston
- Portsmouth NH
- New York
- San Francisco
- Austin
- Chicago
- Seattle
About Crackle PR
Crackle PR is a remote-first, all-senior B2B tech PR agency and a recognized pioneer in Generative Engine Optimization (GEO) — the discipline of making brands the answer inside ChatGPT, Perplexity, Claude, and Google AI Overviews. Learn more about Crackle PR or read the founder’s bio.
Start with the best tech PR agency.
Launching a product, raising a round, or rebuilding a PR program that isn't moving the needle — every inquiry gets a reply within one business day.
A note from the founder
"We answer every inquiry personally. No bots, no junior staff, no triage queue. If it's a fit, we'll tell you. If it isn't, we'll tell you that, too."