What Are AI Crawlers? GPTBot, OAI-SearchBot, ClaudeBot and More
AI crawlers are automated programs that fetch web pages for AI companies. They fall into three groups: training bots such as GPTBot and ClaudeBot, search bots such as OAI-SearchBot and PerplexityBot, and user-triggered bots such as ChatGPT-User and Claude-User. Each company documents them separately, so allowing one does not allow the others 123.
- Three jobs: training a model, building a search index, and fetching a page because a person asked.
- Blocking a training bot does not remove you from AI search. OpenAI says OAI-SearchBot is the one that controls ChatGPT search results 1.
- robots.txt rules may not apply to user-triggered bots, per OpenAI, and Perplexity says Perplexity-User generally ignores them 13.
- robots.txt is a request, not a lock. The standard says its rules are not a form of access authorization 6.
What are AI crawlers?
An AI crawler is a bot that visits web pages on behalf of an AI company. It reads your page and sends the content back, either to be stored or to answer a question. Your server log sees it as a visitor with a name, called a user agent.
The important point for an owner is that AI companies run several bots, not one. Each has its own name, its own job, and its own on-off switch in a file called robots.txt, the plain-text file at the root of your site that tells bots where they may go.
That means a single decision such as "block AI" is too blunt. You may want to be found in ChatGPT answers while keeping your content out of model training. The bots make that split possible.
What are the three types of AI crawler?
Training bots collect content that may be used to build future models. Search bots build an index so your pages can appear and be linked in an AI search answer. User-triggered bots fetch one page at the moment a person asks about it.
OpenAI documents exactly this split. OAI-SearchBot surfaces sites in ChatGPT search, GPTBot crawls content that may be used to train its foundation models, and ChatGPT-User handles actions that a user starts 1. Anthropic uses the same three-part pattern under different names, and Perplexity documents a search bot and a user-triggered bot 23.
Checked against each vendor's documentation on 20 October 2026. Bot names and rules change, so re-check before you edit robots.txt.
| Bot | Company | Job | What blocking it does |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search | Your site will not be shown in ChatGPT search answers 1 |
| GPTBot | OpenAI | Training | Signals your content should not be used to train its models 1 |
| ChatGPT-User | OpenAI | User-triggered | Robots.txt rules may not apply, and it does not decide Search inclusion 1 |
| ClaudeBot | Anthropic | Training | Signals future material should be excluded from training sets 2 |
| Claude-SearchBot | Anthropic | Search | Prevents indexing for search, which may reduce visibility in results 2 |
| Claude-User | Anthropic | User-triggered | Anthropic says blocking it may reduce your visibility in answers to user questions 2 |
| PerplexityBot | Perplexity | Search | Perplexity recommends allowing it so your site can appear in its results 3 |
| Perplexity-User | Perplexity | User-triggered | Generally ignores robots.txt because a person asked for the page 3 |
| Google-Extended | Control token for Gemini training and grounding | Does not affect inclusion in Google Search 4 |
What did an AI assistant look for when we asked about these bots?
On 20 October 2026 we gave this question to Claude with a web search tool. It ran two searches: "GPTBot OAI-SearchBot ChatGPT-User difference" and "ClaudeBot PerplexityBot user agents robots.txt". Both name the bots and ask about differences or robots.txt, so this page answers both directly.
This is one run on one day. It tells you which angles the tool chose, not how every assistant behaves. Check your own buyers' questions with our Query Fan-Out Explorer.
Which AI crawlers should I allow?
If you want leads from AI search, allow the search and user-triggered bots. OpenAI recommends allowing OAI-SearchBot, and Perplexity recommends allowing PerplexityBot, so your site can appear in their results 13. Blocking Anthropic's search or user bots may reduce your visibility in Claude answers 2.
Training bots are a business choice. Allowing them may help models learn about your brand over time, though we found no study that proves this. Blocking them keeps your text out of training sets. Neither choice removes you from search, because search bots are separate.
One detail from OpenAI: if you allow both OAI-SearchBot and GPTBot, it may use the results of one crawl for both purposes 1. If you want to keep training off, disallow GPTBot explicitly. Our guide to blocking AI training but still appearing in AI search has the steps.
- User-agent: OAI-SearchBot
- Allow: /
- User-agent: ChatGPT-User
- Allow: /
- User-agent: PerplexityBot
- Allow: /
- User-agent: Claude-SearchBot
- Allow: /
- User-agent: Claude-User
- Allow: /
- User-agent: GPTBot
- Disallow: /
- User-agent: ClaudeBot
- Disallow: /
Can robots.txt really control AI bots?
Only partly. The robots exclusion standard says its rules are not a form of access authorization and are no substitute for valid content security measures 6. A well-behaved bot honours them. A badly behaved one can ignore them.
The user-triggered bots show the gap. OpenAI says robots.txt rules may not apply to ChatGPT-User because users start those actions, and Perplexity says the same of Perplexity-User 13. Anthropic says its bots honour robots.txt and that blocking its IP addresses may not work, because the bot then cannot read your robots.txt file 2.
There is also a dispute on record. On 4 August 2025 Cloudflare reported that Perplexity used undeclared crawlers to get around no-crawl rules, citing 20 to 25 million daily requests from the declared Perplexity-User agent and 3 to 6 million from a generic browser-like agent. The report does not state whether Perplexity disputed it 5. Treat it as one vendor's claim, not a settled finding.
If content must stay private, put it behind a login. Robots.txt is for courtesy and preference, not secrecy.
How do I see which AI bots visit my site?
Search your server access log for the bot names and count requests per address. User-agent strings can be faked, so confirm suspicious visitors against the vendors' published IP lists 123.
- grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' access.log | awk '{print $1}' | sort | uniq -c | sort -rn
Can AI crawlers read JavaScript pages?
Often not, according to one study we could find. Vercel's analysis of its network traffic, published 17 December 2024, found that none of the major AI crawlers it tested rendered JavaScript. These included OpenAI's crawlers, ClaudeBot and PerplexityBot. It listed Gemini and AppleBot as able to render 7.
That study is almost two years old, so behaviour may have changed. The safe approach is to make sure your key text is in the HTML your server sends. Our guide on making your site server-side rendered covers this, and the AI Crawler Checker shows what each bot can read today.
How is Google different from the others?
Google splits its controls differently. Its crawler documentation describes Google-Extended as a standalone token you place in robots.txt. It covers whether content Google crawls may be used for future Gemini model training and for grounding in Gemini Apps, and it does not affect a site's inclusion in Google Search 4.
Google-Extended has no separate user-agent string. Google crawls with its existing agents and uses the token only for control 4. So there is no way to see Google-Extended in your logs, unlike GPTBot.
What an AI assistant searches when it answers this
We gave the question below to AI search tools and recorded the web searches they ran behind the scenes. This is called query fan-out. Each phrase is a page you could publish to be found and cited.
“what are GPTBot OAI-SearchBot ChatGPT-User ClaudeBot PerplexityBot”
The searches it ran- GPTBot OAI-SearchBot ChatGPT-User difference
- ClaudeBot PerplexityBot user agents robots.txt
Frequently asked questions
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot crawls content that may be used to train OpenAI models. OAI-SearchBot surfaces sites in ChatGPT search. ChatGPT-User runs when a person triggers an action, and it does not crawl automatically 1. Each has its own robots.txt setting.
Does blocking GPTBot remove me from ChatGPT search?
No. OpenAI says disallowing GPTBot signals that content should not be used for training, while OAI-SearchBot controls ChatGPT search results 1. Sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers.
Which Claude bot should I allow?
Allow Claude-SearchBot and Claude-User if you want visibility in Claude search and user questions. Anthropic says blocking them may reduce that visibility 2. ClaudeBot is the training bot, so blocking it only signals you want future content excluded from training.
Do AI bots obey robots.txt?
Anthropic says its bots, including Claude-User, honour robots.txt 2. OpenAI documents separate robots.txt settings for OAI-SearchBot and GPTBot but says rules may not apply to ChatGPT-User 1. Perplexity documents settings for PerplexityBot but says Perplexity-User generally ignores them 3. The standard itself says robots.txt is not access control 6.
Does Google-Extended affect my Google rankings?
No. Google says Google-Extended does not impact inclusion in Google Search and is not a ranking signal 4. It only controls use of crawled content for Gemini training and grounding.
Sources
- Overview of OpenAI Crawlers · OpenAIOpened October 20, 2026. Supports: OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot purposes; recommendation to allow OAI-SearchBot; robots.txt may not apply to ChatGPT-User; independent settings; about 24 hours for changes; IP lists.
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic (Claude Help Center)Opened October 20, 2026. Supports: ClaudeBot, Claude-User and Claude-SearchBot roles and the effect of blocking each; robots.txt and Crawl-delay support; IP-blocking caveat; published IP list.
- Perplexity Crawlers · PerplexityOpened October 20, 2026. Supports: PerplexityBot surfaces sites in search and is not used for foundation model training; Perplexity-User generally ignores robots.txt; changes take up to 24 hours; IP lists.
- Google's common crawlers · Google Search CentralOpened October 20, 2026. Supports: Google-Extended is a standalone robots.txt token for Gemini training and grounding, has no separate user-agent string, and does not affect inclusion in Google Search.
- Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives · CloudflareOpened October 20, 2026. Supports: Cloudflare's August 2025 claim of undeclared crawling by Perplexity, with request volumes; the page does not state whether Perplexity disputed it.
- RFC 9309: Robots Exclusion Protocol · IETF (RFC Editor)Opened October 20, 2026. Supports: Robots.txt rules are not a form of access authorization; crawlers combine matching user-agent groups and fall back to the wildcard group.
- The rise of the AI crawler · Vercel (with MERJ)Opened October 20, 2026. Supports: None of the major AI crawlers tested rendered JavaScript, with Gemini and AppleBot as exceptions. Published 17 December 2024.