Skip to content
Chalk Labs / AI SEO
Comparison

AI Training Crawlers vs AI Search Crawlers: Which to Allow?

By Rahil Jain, founder of Chalk LabsUpdated How we research

The short answer

AI training crawlers collect web content that may be used to build future models. AI search crawlers find and fetch pages so an assistant can answer a question today and link to you. OpenAI and Anthropic publish separate bots for each job, and Google uses a token for training, so a business can usually stay visible in AI search while refusing training.

  • OpenAI and Anthropic publish separate bots for training, search and user-triggered visits. Perplexity lists only search and user-triggered bots. Google controls training with a token, Google-Extended.
  • OpenAI says its training and search bots are separate settings, and Google says Google-Extended does not affect Search. Blocking the search bot removes you from that company's answers.
  • User-triggered fetchers may ignore robots.txt, so it is not a privacy tool.
  • Training crawling is far larger than search crawling and sends back few visitors. Decide on policy, not on traffic.

What is the difference between training crawlers and search crawlers?

A training crawler gathers content to help build or improve a model. A search crawler gathers content so an assistant can retrieve and cite it when someone asks a question. The first is about the model's future knowledge. The second is about your visibility today.

OpenAI documents the split most clearly. GPTBot crawls content that may be used in training its foundation models, while OAI-SearchBot surfaces sites in ChatGPT search, and each setting is independent 1. Anthropic does the same with ClaudeBot for training and Claude-SearchBot for improving search results 2.

A third group exists: bots that act when a person asks. ChatGPT-User, Claude-User and Perplexity-User fetch a page on demand 123. We cover the OpenAI trio in GPTBot vs OAI-SearchBot vs ChatGPT-User.

Which one fits you?

A short quiz to match a crawler policy to your goals. It is a rule of thumb, not legal advice, and it applies to all AI companies at once.

Allow search, refuse training

Let AI search bots fetch you, and disallow the training bots from each company.

Best when
  • You want to be cited and recommended.
  • You publish original research or paid material.
  • You want a documented training opt-out.

Allow everything

Leave all AI bots free to fetch public pages.

Best when
  • Your content is meant to spread widely.
  • You see brand benefit in models knowing you.
  • You have no licensing concerns.

Block AI bots

Disallow training and search bots, and protect private pages with logins.

Best when
  • Your content is the product.
  • You are not trying to win AI recommendations.
  • You can accept losing AI search visibility.

A short quiz

0 of 3 answered

Question 1 of 3How important is being recommended by AI assistants to your sales?
Question 2 of 3Is your public content something you sell or license?
Question 3 of 3How do you feel about future models learning from your pages?

Which bot does which job at each company?

The matrix lists the named bots from each company's own documentation. Perplexity says PerplexityBot is not used to crawl content for foundation models, so it lists no separate training bot 3. Google handles training through a control token, not a separate crawler 4.

Training, search and user-triggered bots by company
CompanyTrainingSearchUser-triggered
OpenAIGPTBotOAI-SearchBotChatGPT-User
AnthropicClaudeBotClaude-SearchBotClaude-User
PerplexityNone listedPerplexityBotPerplexity-User
GoogleGoogle-Extended (control token)GooglebotUser-triggered fetchers (see Google's separate page)

Why is Google different?

Google ties Search and AI features to the same crawler, Googlebot. Google says its common crawlers always obey robots.txt when crawling automatically, and that Google-Extended is a standalone token controlling use for training future Gemini models and for grounding in Gemini Apps and Vertex AI 4. It adds that Google-Extended does not affect inclusion in Google Search and is not a ranking signal 4.

There is no separate Googlebot for AI Overviews. A page needs to be indexed and eligible to show in Search with a snippet to appear as a supporting link in AI Overviews or AI Mode 5. So blocking Googlebot to avoid AI would also take you out of Google Search. Google lets you limit what is shown with nosnippet, data-nosnippet, max-snippet or noindex instead 5.

How big is the gap between training and search crawling?

Training dominates. Cloudflare's 2025 review found training crawls made up the vast majority of AI crawler traffic, reaching as much as 7 to 8 times search crawling and 32 times user-action crawling at peak 6. Search crawling peaked in mid-March and then fell about 40%, while user-action crawling grew by more than 15 to 21 times from January to early December (the report gives both figures) 6.

The traffic returned to publishers is small. Cloudflare reports crawl-to-refer ratios, which compare pages fetched with visits sent back. Anthropic's reached as much as 500,000:1, OpenAI's 3,700:1 in March, and Perplexity's stayed below 200:1 from September 6. Google ranged from about 3:1 up to a 30:1 peak in April 6.

Read these as signs of imbalance, not as proof that blocking costs or saves you anything. A site with few visits from AI today may still be cited in answers that people read without clicking.

What do you gain and lose by allowing each type?

Allowing search bots buys eligibility to be cited. Anthropic says disabling Claude-SearchBot may reduce your visibility and accuracy in user search results, and disabling Claude-User may reduce visibility in user-directed web search 2. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers 1. The cost of allowing is bandwidth and the chance a competitor is cited instead.

Allowing training bots buys nothing you can measure today. The possible upside is that a future model knows your brand without searching. The possible downside is that your original work is reused with no credit or payment. We found no published study showing that allowing training bots raises citations, so treat the upside as unproven.

Blocking everything is a clean policy for paid or licensed content. Blocking only training is the middle path for most businesses that sell by being found.

Does robots.txt actually stop them?

It stops compliant bots, and the major companies say theirs comply. Anthropic says its bots honour industry-standard robots.txt directives and advises against IP blocking, which can stop them from reading robots.txt at all 2. OpenAI and Perplexity both say user-initiated fetches may ignore robots.txt 13.

Compliance is also contested. In August 2025 Cloudflare alleged that Perplexity used undeclared crawlers with a generic browser user agent to reach sites that had blocked it, and Perplexity has disputed the claim 7. We cannot judge who is right. The practical lesson is that robots.txt is a request, not a lock. Anything truly private belongs behind a login.

What does an allow-search, block-training file look like?

Here is a starting file for a business that wants to be found but refuses training. It uses only the bot names from the companies' own pages 1234. Test it with the AI Crawler Checker before and after you publish.

Allow lines are optional when your file has no broader block, but they make intent clear to the next person who edits it. Changes can take about a day to reach OpenAI's search systems 1.

robots.txt: search allowed, training refused
  • # Training: refuse
  • User-agent: GPTBot
  • Disallow: /
  • User-agent: ClaudeBot
  • Disallow: /
  • User-agent: Google-Extended
  • Disallow: /
  • # Search: allow
  • User-agent: OAI-SearchBot
  • Allow: /
  • User-agent: Claude-SearchBot
  • Allow: /
  • User-agent: PerplexityBot
  • Allow: /

How do you choose a policy for your site?

Start from what your site sells. A business that earns by being recommended, such as a local firm or a B2B service, usually wants search bots on. A publisher that sells access to its writing may block both. The quiz turns this into a short list of questions.

For the step-by-step setup, follow how to block AI training but still appear in AI search and how to allow AI crawlers in robots.txt. For the wider list of bots, see what AI crawlers are.

Frequently asked questions

Will blocking training bots hurt my AI search visibility?

Not by itself, according to the companies. OpenAI treats GPTBot and OAI-SearchBot as independent settings 1. Google says Google-Extended does not affect inclusion in Google Search 4. Anthropic's article does not say whether its search and user bots feed training 2.

Should a small business block AI training crawlers?

For many small businesses the pages are marketing meant to be found, so blocking training may bring little benefit. Block training if you sell original content or have licensing duties. If you do not, the bigger risk is accidentally blocking the search bots that bring you recommendations.

Does Google-Extended control AI Overviews?

No. Google says Google-Extended controls use of crawled content for training future Gemini models and for grounding in Gemini Apps and Vertex AI, and does not affect Search inclusion 4. AI Overviews depend on normal Search eligibility, and you limit them with snippet controls 5.

Is Perplexity a training crawler?

Perplexity says PerplexityBot is designed to surface and link sites in search results and is not used to crawl content for AI foundation models 3. It also says Perplexity-User generally ignores robots.txt because the visit is user-requested 3.

How often should I review my AI crawler rules?

Review them every three to six months, and whenever a company announces a new bot. OpenAI, for example, now lists a fourth agent for ads 1. Keep the file short and commented so the next editor understands each rule.

About the author

Rahil Jain

Rahil Jain is an AI SEO (answer engine and generative engine optimisation) consultant and co-founder of Chalk Labs.

LinkedIn · Method and credentials

Sources

  1. Overview of OpenAI Crawlers · OpenAIOpened October 20, 2026. Supports: GPTBot for training, OAI-SearchBot for ChatGPT search, ChatGPT-User for user actions; settings are independent; robots.txt may not apply to user-initiated actions; about 24 hours to adjust; OAI-AdsBot exists.
  2. Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic SupportOpened October 20, 2026. Supports: ClaudeBot, Claude-User and Claude-SearchBot roles; consequences of blocking; bots honour robots.txt; IP blocking not recommended; article silent on training use of Claude-User and Claude-SearchBot data. Dated April 7, 2026.
  3. Perplexity Crawlers · PerplexityOpened October 20, 2026. Supports: PerplexityBot is for search results, not foundation model crawling; Perplexity-User generally ignores robots.txt.
  4. Google's common crawlers · Google Search CentralOpened October 20, 2026. Supports: Google-Extended is a standalone token for Gemini training and grounding and does not affect Search inclusion or ranking; common crawlers obey robots.txt when crawling automatically. Last updated 2026-10-15.
  5. AI features and your website · Google Search CentralOpened October 20, 2026. Supports: A page must be indexed and snippet-eligible to appear in AI Overviews or AI Mode; nosnippet, data-nosnippet, max-snippet and noindex limit what is shown.
  6. Cloudflare Radar 2025 Year in Review · CloudflareOpened October 20, 2026. Supports: Training crawling up to 7-8x search and 32x user-action crawling at peak; search crawling fell about 40% after mid-March; user-action crawling up over 21x (a section heading says 15x); crawl-to-refer ratios by company.
  7. Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives · CloudflareOpened October 20, 2026. Supports: On 4 Aug 2025 Cloudflare alleged Perplexity used undeclared crawlers to access sites that had blocked it. The page contains no Perplexity response.
How this page was made. Built on Rahil Jain's AI SEO method and drafted with AI research assistance. Every statistic links to the page we read it on (opened October 20, 2026 or nearby). Published October 20, 2026; last updated October 20, 2026. Chalk Labs sells AI SEO services, so we have a commercial interest in this topic. Spotted an error? Tell us and we will fix it.