Skip to content
Chalk Labs / AI SEO
How-to guide

How to Block AI Training but Still Appear in AI Search

By Rahil Jain, founder of Chalk LabsUpdated How we research

The short answer

You can, because the big AI companies run separate bots for training and for search. In robots.txt, disallow the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) and allow the search tokens (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot). Two limits apply: robots.txt is a request, not a lock, and user-triggered fetchers may ignore it.

  • OpenAI, Anthropic, Perplexity, Google and Apple each document separate tokens, so training and search can be set independently.
  • Disallow GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot.
  • Google-Extended does not control Google Search or AI Overviews. Use Googlebot rules and snippet controls for those.
  • robots.txt is a polite request. Fetchers that act on a user's request, such as ChatGPT-User, may not follow it.

Can you really block AI training and still appear in AI search?

Yes, with one honest limit. Each company that publishes crawler rules uses a different token for training than for search. A token is the name a bot gives itself, and you write it on the User-agent line of robots.txt. OpenAI says each of its settings is independent of the others, so a site can allow OAI-SearchBot and disallow GPTBot 1.

The limit is that robots.txt is a request. The protocol states that its rules are not a form of access authorization 5. Well-run crawlers follow it. Nothing forces the rest to.

There is also a trade-off with no reliable data. We hold no study showing whether blocking training bots changes how often a brand is mentioned in AI answers. Treat that as unknown, and decide on principle: do you want your pages used to train models, yes or no?

Do it step by step

0 of 8 done0%

  1. Write down what you want: for example, appear in AI answers, but do not allow model training. Everything else follows from that sentence.

    Tip: If you cannot decide, keep search allowed and revisit training later. Allowing is easier to undo than months of lost visibility.

  2. Visit yourdomain.com/robots.txt for every host you run, including www, blog and docs subdomains. Save a copy so you can roll back.

    Tip: Plugins and CDNs sometimes add AI blocks. Look for lines you did not write.

  3. Paste the template below into your file. It allows the search tokens and disallows the training tokens.

  4. A bot with its own group ignores your wildcard group. Add Disallow lines for /account/, /cart/ and similar paths under every group you create.

    Tip: If the same list appears in five groups, that is normal. It is how the protocol works.

  5. The file must live at the top level, such as https://yourdomain.com/robots.txt. Repeat it for each subdomain you want covered.

    Tip: Fetch the live URL in a browser afterwards to make sure the file is plain text and returns a 200 status.

  6. Run the curl test below. A 403 or a challenge page for a bot you allowed means a firewall rule is overriding your robots.txt.

    Tip: Allow the search bots' published IP ranges if your firewall blocks unknown traffic.

  7. OpenAI and Perplexity both cite up to 24 hours for changes. After that, look for GPTBot, ClaudeBot and the other disallowed tokens in your server logs and confirm they stop requesting pages.

  8. A week later, ask ChatGPT, Perplexity and Claude a question your page answers well, with search on. Note whether your site appears as a source. Repeat monthly.

    Tip: Log the date, tool and result in a simple spreadsheet. One test proves little; a pattern proves more.

Your ticks are saved in this browser only.

Copy-ready templates

robots.txt: search allowed, training disallowed

text
# Search and answer bots: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Applebot
Allow: /
Disallow: /account/
Disallow: /cart/

# Training bots: disallowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including Googlebot
User-agent: *
Disallow: /account/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

Replace /account/ and /cart/ with your own private paths and the Sitemap URL with yours. Googlebot has no group of its own here, so it follows the wildcard group and stays allowed. Add a CCBot group only if you want to block Common Crawl.

Curl test for firewall blocks

bash
for bot in "OAI-SearchBot" "PerplexityBot" "Claude-SearchBot" "GPTBot"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" -A "$bot" https://www.example.com/pricing/)
  echo "$bot -> HTTP $code"
done

This sends a request with each bot's name as the user agent. It tests your firewall, not your robots.txt. Real bots also use their own IP ranges, so a pass here is a good sign, not proof.

Policy note to keep with your files

markdown
## AI crawler policy (review every quarter)
- Goal: appear in AI search answers; no model training use.
- Allowed: OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, Applebot.
- Disallowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended.
- Known gap: user-triggered fetchers (ChatGPT-User, Perplexity-User) may ignore robots.txt.
- Last reviewed: YYYY-MM-DD   Owner: NAME

Short, dated and owned. Whoever edits robots.txt next year will thank you.

Which token controls training and which controls search?

The table lists what each company documents. OpenAI says OAI-SearchBot surfaces sites in ChatGPT search, and sites opted out of it will not be shown in ChatGPT search answers 1. It says GPTBot crawls content that may be used to train its models 1. ChatGPT-User acts on a user's request, and OpenAI says robots.txt rules may not apply to it 1.

Anthropic lists three bots. ClaudeBot collects content that may contribute to training. Claude-User fetches pages when a person asks Claude a question. Claude-SearchBot helps improve search results 2. Perplexity says PerplexityBot surfaces sites in its results and is not used to crawl content for foundation models 4.

Suggested setting if you want search visibility but no training use
TokenCompanyDocumented jobSuggested setting
OAI-SearchBotOpenAISurfaces sites in ChatGPT search 1Allow
GPTBotOpenAICrawls content that may train models 1Disallow
ChatGPT-UserOpenAIVisits pages on a user's request 1Leave allowed; robots.txt may not apply
Claude-SearchBotAnthropicImproves search result quality 2Allow
Claude-UserAnthropicFetches pages when a user asks 2Allow
ClaudeBotAnthropicCollects content that may contribute to training 2Disallow
PerplexityBotPerplexitySurfaces and links sites in results 4Allow
Perplexity-UserPerplexityFetches a page when a user asks 4Generally ignores robots.txt
GooglebotGoogleCrawls for Search and its features 3Allow
Google-ExtendedGoogleControls Gemini training and grounding use 3Disallow
ApplebotApplePowers search in Spotlight, Siri and Safari 7Allow
Applebot-ExtendedAppleControls training use of Applebot data 7Disallow
CCBotCommon CrawlFeeds an open repository of crawl data 8Your call

Does blocking Google-Extended remove you from Google AI Overviews?

No. Google says Google-Extended is a standalone token that controls whether crawled content may be used to train future Gemini models and for grounding in some Gemini products. It also says the token does not affect inclusion in Google Search 3.

Google's page on AI features says a page needs to be indexed and eligible to show with a snippet to appear in AI Overviews and AI Mode, with no additional technical requirements 6. The controls it names are Googlebot rules, nosnippet, data-nosnippet, max-snippet and noindex 6. That page does not say Google-Extended affects AI Overviews.

So the split is simple. To keep AI Overviews, keep Googlebot allowed and avoid snippet limits. To opt out of Gemini training use, disallow Google-Extended. Never use a blanket block on Googlebot as a shortcut.

Why do some robots.txt rules not work as expected?

The usual cause is grouping. A crawler follows the group that names it. If none match, it uses the wildcard group marked with an asterisk 5. So once GPTBot has its own group, your private-folder rules under the asterisk no longer apply to it. Repeat them.

When several rules match a path, the most specific one wins, meaning the one with the most characters 5. Crawlers may cache the file, but the protocol says they should not use a cached copy for more than 24 hours 5.

Rules are also per host. Anthropic says to add its block to the top-level robots.txt and repeat it for each subdomain 2. A rule on www does not cover blog or docs subdomains.

What can robots.txt not stop?

It cannot stop a visit that a person triggers. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user starts the action 1. Perplexity says Perplexity-User generally ignores robots.txt for the same reason 4.

It also cannot hide pages. The protocol says robots.txt is not a substitute for valid content security 5. If a page must stay private, put it behind a login.

Do not rely on IP blocking either. Anthropic warns that blocking IP addresses may not work correctly or persistently and recommends robots.txt for opting out, because an IP block can stop the bot reading your robots.txt at all 2.

What does a finished file look like?

Here is a worked example for a site that wants search visibility and no training use, with a private /account/ folder. Notice that the private-folder rule is repeated in every group that still allows crawling. The training group blocks the whole site, so it needs no extra lines.

Before and after
  • Before: User-agent: * / Disallow: /account/ (every bot, training and search, is treated the same).
  • After, allowed group: User-agent: OAI-SearchBot, Claude-SearchBot, PerplexityBot / Allow: / / Disallow: /account/
  • After, training group: User-agent: GPTBot, ClaudeBot, Google-Extended / Disallow: /
  • After, wildcard group: User-agent: * / Disallow: /account/
  • Result to check: the search bots can fetch /pricing/ but not /account/. GPTBot, ClaudeBot and Google-Extended cannot fetch anything.

How do you check that it worked?

Check three things. First, read your live file at yourdomain.com/robots.txt. Second, run the AI crawler checker to see how each named bot is treated. Third, look for firewall or CDN blocks, which robots.txt cannot reveal.

Then wait. OpenAI and Perplexity both say changes can take up to 24 hours to take effect 14. After that, check your server logs for the bots you disallowed. Server log monitoring shows whether they stopped.

For the wider picture, read AI training crawlers vs AI search crawlers and the guide to allowing AI crawlers in robots.txt.

When should you not block training bots?

In our view, skip the block if your goal is to be as widely known as possible and your content is meant to be public and reused. Open documentation, public research and free tools often fit this case.

Block if your content is the product: paid reports, licensed data, original journalism or member-only guides. Then training use competes with what you sell. Whichever you choose, write the decision down and review it every quarter, because the token lists change. Google's crawler page was last updated on 15 October 2026 3.

Frequently asked questions

Will blocking GPTBot remove me from ChatGPT search?

No, according to OpenAI. GPTBot is the training crawler, and OpenAI says each robots.txt setting is independent of the others. Visibility in ChatGPT search depends on OAI-SearchBot. Keep that one allowed, and confirm the change after about 24 hours.

Does Google-Extended affect my Google rankings?

Google says it does not. Google-Extended is a standalone control for Gemini training and grounding, and Google states it does not affect inclusion in Google Search or act as a ranking signal. AI Overviews are governed by Googlebot rules and snippet controls instead.

Do AI companies actually obey these rules?

Anthropic says its bots honour robots.txt do-not-crawl directives, and Google says its common crawlers always obey automatic-crawl rules. Fetchers started by a user, such as ChatGPT-User and Perplexity-User, may not. robots.txt is a request, so use a login for anything private.

Should I block Claude-User as well as ClaudeBot?

Usually not. Anthropic says blocking Claude-User stops retrieval when someone asks Claude about your content, which may reduce your visibility there. ClaudeBot is the training one. Block that, and keep Claude-User and Claude-SearchBot allowed if you want to be found.

Is it too late to opt out if my content was already crawled?

The documents we read describe future crawling and future training use only. They do not say content already collected will be removed. A robots.txt rule signals your choice from now on. For past use, check each company's own data-removal process.

About the author

Rahil Jain

Rahil Jain is an AI SEO (answer engine and generative engine optimisation) consultant and co-founder of Chalk Labs.

LinkedIn · Method and credentials

Sources

  1. Overview of OpenAI Crawlers (Bots) · OpenAIOpened October 20, 2026. Supports: OAI-SearchBot surfaces sites in ChatGPT search; GPTBot is for training; ChatGPT-User robots.txt rules may not apply; settings are independent; about 24 hours to take effect; one crawl may serve both uses.
  2. Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic (Claude Help Center)Opened October 20, 2026. Supports: ClaudeBot, Claude-User and Claude-SearchBot roles and the effect of blocking each; robots.txt honoured; repeat per subdomain; IP blocking is not a reliable opt-out.
  3. Google's common crawlers · Google Search CentralOpened October 20, 2026. Supports: Google-Extended is a standalone token for Gemini training and grounding that does not affect Search; Googlebot rules apply to Search; common crawlers obey robots.txt; last updated 2026-10-15.
  4. Perplexity Crawlers · PerplexityOpened October 20, 2026. Supports: PerplexityBot surfaces sites in results and is not used for foundation model crawling; Perplexity-User generally ignores robots.txt; changes can take up to 24 hours.
  5. RFC 9309: Robots Exclusion Protocol · IETF (RFC Editor)Opened October 20, 2026. Supports: Rules are not access authorization; grouping and wildcard fallback; most specific rule wins; caching should not exceed 24 hours.
  6. AI features and your website · Google Search CentralOpened October 20, 2026. Supports: Inclusion in AI Overviews and AI Mode requires indexing and snippet eligibility with no additional technical requirements; named controls are nosnippet, data-nosnippet, max-snippet and noindex.
  7. About Applebot · Apple SupportOpened October 20, 2026. Supports: Applebot data powers search in Spotlight, Siri and Safari; Applebot-Extended does not crawl and governs training use; blocking it does not remove a site from search.
  8. CCBot · Common CrawlOpened October 20, 2026. Supports: CCBot is Common Crawl's crawler; Common Crawl maintains an open repository of crawl data; how to block it in robots.txt.
How this page was made. Built on Rahil Jain's AI SEO method and drafted with AI research assistance. Every statistic links to the page we read it on (opened October 20, 2026 or nearby). Published October 20, 2026; last updated October 20, 2026. Chalk Labs sells AI SEO services, so we have a commercial interest in this topic. Spotted an error? Tell us and we will fix it.