Skip to content
Chalk Labs / AI SEO
How-to guide

How to Get Cited by Claude

By Rahil Jain, founder of Chalk LabsUpdated How we research

The short answer

Claude searches the web when a question needs current information and cites the pages it uses. To be cited, allow Claude-SearchBot and Claude-User in robots.txt, keep your pages indexable on the open web, write sections that answer one question each, and test your real buyer prompts every month. Anthropic publishes no ranking-factor list and does not name its search provider.

  • Anthropic runs three bots. Blocking Claude-SearchBot stops indexing for search, and blocking Claude-User stops fetching on a user's request.
  • Anthropic does not name the search index behind Claude in the pages we read, so broad indexing is the safe aim.
  • Claude sent about 102,000 visits in Ahrefs' March 2026 sample, so measure mentions first.
  • Semrush Prompt Tracking does not support Claude, so manual prompt testing is the practical method.

How does Claude decide when to search and what to cite?

Claude searches when a question depends on current or changing information, such as recent news, prices, or facts about a specific organisation 1. For stable knowledge, such as maths or established science, it answers directly without searching 1. Your page can only be cited on questions where Claude chooses to search.

Anthropic's help centre says every response that uses web search includes citations so users can check sources 2. In the developer web search tool, Claude can run searches several times in one request, and simple factual queries typically use one to three searches 1. That means one prompt can pull in several pages, and each page has to answer a narrow sub-question.

Anthropic publishes no list of ranking factors for Claude, and we found none in the pages we read. Everything below combines Anthropic's documented crawler rules with outside studies. We mark which is which.

Do it step by step

0 of 8 done0%

  1. Write down one line per bot: ClaudeBot (training), Claude-User (user requests), Claude-SearchBot (search indexing). To be cited, allow the last two 3. Decide on ClaudeBot separately, since it relates to training.

    Tip: Keep the decision in your team wiki so nobody undoes it later.

  2. Add the Claude rules from the robots.txt template to the top-level file on every subdomain you want covered 3. Keep your private-path rules inside the Claude group, since a bot with its own group ignores the general group.

    Tip: Keep truly private pages behind a login, not just in robots.txt.

  3. Open yourdomain.com/robots.txt in a browser and confirm the new lines appear. Then fetch it with the user agent from the bash template to be sure your CDN does not serve it a challenge page.

    Tip: A 403 or a CAPTCHA page for bots is a common hidden block.

  4. Make sure your bot protection does not block Anthropic's user agents. Do not rely on IP rules alone, because Anthropic warns IP blocking may not work correctly 3. Use the published list in claude.com/crawling/bots.json only to confirm a hit came from Anthropic.

  5. Submit your sitemap in Google Search Console and Bing Webmaster Tools. Add IndexNow so Bing, Naver, Seznam.cz, Yandex and Yep hear about new URLs 9. Anthropic does not name its search index, so cover the main ones.

  6. Use Claude with web search on. Log each prompt, whether you are named, whether your domain is linked, and who is cited instead, using the prompt log template. Run each three times 5.

    Tip: Use a fresh chat each time so earlier answers do not colour the next.

  7. For each prompt where a rival is cited, rewrite your matching page with a question heading, a first-sentence answer, a named source for each number and a date. For local firms, also fix your review-site profiles, which Anthropic's engine cited heavily in Yext's study 5.

  8. After a week, run the log commands and look for Claude-SearchBot and Claude-User hits with status 200. Repeat the prompt test monthly and add rows instead of overwriting.

    Tip: No hits on a small new site is normal. Anthropic gives no crawl schedule.

Your ticks are saved in this browser only.

Copy-ready templates

robots.txt: allow Claude's search bots, keep training out

text
# Search and user-request bots: allow so Claude can index and cite you
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /admin/
Allow: /

# Training crawler: optional. Remove this block to allow training too.
User-agent: ClaudeBot
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin/

Replace /admin/ with your own private paths. Anthropic says its bots honour robots.txt and you should repeat the rules on each subdomain . Anthropic does not say whether the two search bots feed training .

Bash: test robots.txt and read your logs

bash
SITE=https://www.example.com
LOG=access.log   # your log path

# 1. Fetch robots.txt as Claude-SearchBot would identify itself
curl -s -o /dev/null -w "robots.txt status: %{http_code}\n" -A "Claude-SearchBot" "$SITE/robots.txt"

# 2. How many hits per Anthropic bot?
for b in ClaudeBot Claude-User Claude-SearchBot; do
  printf "%-18s %s\n" "$b" "$(grep -c "$b" "$LOG")"
done

# 3. Status codes the search bot received (field 9, combined log format)
grep 'Claude-SearchBot' "$LOG" | awk '{print $9}' | sort | uniq -c | sort -rn

# 4. Anthropic's published address list
curl -s https://claude.com/crawling/bots.json | grep -o '"ipv4Prefix": "[^"]*"' | head -20

Set SITE and LOG first. Step 4 prints the first 20 of Anthropic's listed prefixes . Some are blocks such as /22, so compare by network, not by exact match.

Prompt log for Claude (spreadsheet columns)

markdown
| Date | Prompt | Run | Web search used? | Brand named? | Domain linked? | Other sources cited | Page to fix |
|---|---|---|---|---|---|---|---|
| 2026-10-20 | best [category] in [city] | 1 | yes | no | no | review-site.com, rival.com | /services/[category] |
| 2026-10-20 | what does [service] cost | 1 | yes | yes | no | rival.com | /pricing |

Claude searches only when it judges the question needs current information , so record whether it did. Replace brackets with your own details.

Which Anthropic bots should you allow?

Allow Claude-SearchBot and Claude-User if you want to be cited. Anthropic's help article lists three bots. ClaudeBot collects web content that may contribute to model training. Claude-User accesses sites when people ask Claude a question. Claude-SearchBot crawls the web to improve search result quality 3.

Anthropic says blocking Claude-User prevents its system retrieving your content in response to a user query, and blocking Claude-SearchBot prevents indexing your content for search 3. Either block may reduce your visibility in Claude's answers 3. Blocking ClaudeBot signals that future material should be excluded from training datasets 3.

So the choice is separable. You can allow the two search-related bots and block ClaudeBot. The article does not say whether Claude-User or Claude-SearchBot data is used for training, so read it again before you decide 3.

Anthropic's three bots and what blocking each does (from Anthropic's help article)
BotWhat Anthropic says it doesEffect of blocking it
ClaudeBotCollects web content that may contribute to model trainingSignals that future materials be excluded from training datasets
Claude-UserFetches pages when a person asks Claude a questionStops retrieval of your content for user queries; may reduce visibility
Claude-SearchBotCrawls the web to improve search result qualityStops indexing for search; may reduce visibility and accuracy in results

Does Claude obey robots.txt, and should you use IP rules?

Anthropic says its bots honour industry-standard robots.txt directives, and it supports the non-standard Crawl-delay line 3. Put the rules in the top-level robots.txt, once for every subdomain 3.

Use robots.txt, not IP blocking, as your main control. Anthropic warns that blocking IP addresses may not work correctly or reliably guarantee an opt-out, because it can stop Anthropic reading your robots.txt 3. Anthropic publishes a crawler address list at claude.com/crawling/bots.json 3.

That file lists IPv4 prefixes only, with no label per bot, when we opened it on 20 October 2026. It can tell you a hit came from Anthropic. It cannot tell you which of the three bots made it. Use the user agent for that.

Which search index does Claude use, and does it matter?

Anthropic does not name the search provider behind Claude.ai in the pages we read. Some online guides say it is Brave Search. We could not confirm that from an Anthropic page. Brave's own API page says the world's biggest AI labs use its search, including a top-three foundation model, and it does not name which one 4.

So treat the Brave claim as unproven. The practical advice does not change much: be crawlable, indexable and easy to find on the open web. Submit your sitemap in Google Search Console and Bing Webmaster Tools, and use IndexNow to tell participating engines about new and changed URLs, including Bing, Naver, Seznam.cz, Yandex and Yep 9.

Google is not an IndexNow participant 9. If the index behind Claude changes, broad coverage protects you. If you find a primary source that names the provider, tell us and we will update this page.

What kinds of sources does Claude cite?

We have limited evidence, and it comes from local-business research. Yext compared Gemini, OpenAI, Perplexity and Anthropic on 491,972 local query cases from February to August 2026. It reports that Anthropic cited review sites far more than the other engines did 5.

The four engines named the same top local business in only 4% of cases 5. When all four agreed, the winning business's own website was cited in 91% of cases 5. Yext sells location-data software, and this study covers local queries only.

Our reading: for local and service businesses, your own site plus your review-site profiles are the two places to fix first. For other topics, we found no Claude-specific citation study with published figures, so test your own prompts.

How should you write pages for Claude to quote?

Put the answer first and keep each section self-contained. Because Claude can run several searches per prompt 1, a page that cleanly answers one narrow question is more useful than one that touches ten. Give each section a question heading, an answer in the first sentence, and one dated, sourced fact.

Our guide to writing content AI engines quote covers the pattern, and the free Answer Block Checker tests an opening paragraph. For local businesses, see optimising a local business for AI search.

Developers can limit Claude's search to chosen domains or exclude domains, and organisation admins can restrict which domains it searches 1. This is outside your control, but it is a reason to publish on a domain you own and keep the facts correct.

What did an AI assistant search for when we asked this question?

On 20 October 2026 we asked a Claude agent with web search to research 'how does Claude web search choose sources and how can I be cited'. It ran three searches: how Claude web search selects sources, whether Claude citations depend on Brave Search's index, and whether to allow ClaudeBot and Claude-SearchBot in robots.txt.

That is one run, so it shows what a Claude agent chose to look up, not a rule. It does match the three questions most site owners ask: how are sources chosen, which index, and which bots.

How do you check whether Claude cites you?

Ask Claude with web search on and read the citations. Run your ten buyer prompts, record whether your brand is named and your domain linked, and note who is cited instead. Repeat each prompt a few times, because engines often disagree: in Yext's data, four engines agreed on the top local pick only 4% of the time 5.

Automated tools lag here. Semrush says its Prompt Tracking supports ChatGPT, Google AI Mode and Gemini, and that Claude and Perplexity are not currently supported 7. So a spreadsheet is the realistic method for Claude today.

Expect small visits. Ahrefs counted about 3.5 million AI chatbot visitors in March 2026, 0.28% of total traffic across 74,752 sites, with Claude at about 102,000 6. That is roughly 3% of the AI total by our arithmetic. StatCounter shows Claude at 0.95% of worldwide AI chatbot share in September 2026, but does not explain its method 8.

When is Claude not worth the effort?

If your buyers do not use Claude, spend your time elsewhere first. Claude's share is small in the figures we found 68. Setting up the two bots takes minutes, so do that regardless, but do not build a Claude-only content plan.

Pages that help Claude usually help other engines too. Fix crawl access, then answer pages, then measurement. A Claude-specific project only makes sense if your customers are developers or heavy AI users, and you can show it from your own analytics or sales calls.

What an AI assistant searches when it answers this

We gave the question below to AI search tools and recorded the web searches they ran behind the scenes. This is called query fan-out. Each phrase is a page you could publish to be found and cited.

The prompt

“how does Claude web search choose sources and how can I be cited”

The searches it ran
  • how Claude web search selects sources
  • Claude citations Brave Search index optimization
  • ClaudeBot Claude-SearchBot robots.txt allow

Observed October 20, 2026 using Claude web search. Claude agents were told to run 6 to 10 searches, so that count is our setting, not Claude's own choice. Assistants differ and change often, so treat this as a sample, not a rule. See how we collect it.

Frequently asked questions

Do I need to allow ClaudeBot to be cited?

Not necessarily. Anthropic describes ClaudeBot as the training crawler, and says blocking Claude-SearchBot stops indexing for search while blocking Claude-User stops retrieval on user requests 3. Allow the last two for citations. The article does not say whether those two feed training.

Can I pay or submit my site to appear in Claude?

We found no submission form or paid placement in Anthropic's documentation. The developer docs describe a web search tool with citations, domain filters and per-search pricing for API users, which concerns developers rather than site owners 1. Focus on access and answer quality.

Does Claude use Brave Search?

We could not confirm it. Anthropic's pages we read do not name a provider, and Brave's page does not name which AI lab it serves 4. Be indexable broadly rather than betting on one engine.

How do I know if Claude visited my site?

Search your server logs for the user agent names Claude-SearchBot, Claude-User and ClaudeBot. Anthropic also publishes an IP address list to confirm a hit came from Anthropic 3. The list is not labelled by bot, so the user agent identifies which bot.

Is there a tool that tracks Claude citations?

Semrush says Claude is not currently supported in its Prompt Tracking 7. Other vendors may differ, and we did not test them. A spreadsheet with ten prompts and monthly runs works. See how to build a prompt list.

About the author

Rahil Jain

Rahil Jain is an AI SEO (answer engine and generative engine optimisation) consultant and co-founder of Chalk Labs.

LinkedIn · Method and credentials

Sources

  1. Web search tool · Anthropic (Claude Docs)Opened October 20, 2026. Supports: Claude searches for current or changing information; may search several times; citations always enabled; domain allow and block lists; org admins can restrict domains.
  2. Enabling and using web search · Anthropic (Claude Help Center)Opened October 20, 2026. Supports: Every response includes citations; Claude searches the web when it helps.
  3. Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic (Claude Help Center)Opened October 20, 2026. Supports: Three bots and effects of blocking each; robots.txt and Crawl-delay; IP blocking warning; bots.json address list (article dated 7 April 2026).
  4. Brave Search API · BraveOpened October 20, 2026. Supports: Brave says major AI labs including an unnamed top-three foundation model use its search; Anthropic is not named.
  5. Four AIs, one question, four different answers · Yext ResearchOpened October 20, 2026. Supports: 491,972 local query cases, Feb to Aug 2026; 4% same top business; Anthropic cited review sites far more; own site cited in 91% of unanimous cases.
  6. AI Chatbot Traffic: What It Is, and How to Get More · AhrefsOpened October 20, 2026. Supports: March 2026: about 3.5 million AI chatbot visitors, 0.28% of traffic, 74,752 sites; Claude about 102,000 (published 15 May 2026).
  7. How to build an AI visibility dashboard with Semrush · SemrushOpened October 20, 2026. Supports: Prompt Tracking supports ChatGPT, Google AI Mode and Gemini; Claude and Perplexity not supported (12 October 2026).
  8. AI Chatbot Market Share Worldwide · StatCounter Global StatsOpened October 20, 2026. Supports: September 2026: Claude 0.95% worldwide AI chatbot share; method not explained on the page.
  9. IndexNow documentation · IndexNow.orgOpened October 20, 2026. Supports: IndexNow lets sites notify participating engines of new or changed URLs; Bing, Naver, Seznam.cz, Yandex, Yep participate; Google does not.
How this page was made. Built on Rahil Jain's AI SEO method and drafted with AI research assistance. Every statistic links to the page we read it on (opened October 20, 2026 or nearby). Published October 20, 2026; last updated October 20, 2026. Chalk Labs sells AI SEO services, so we have a commercial interest in this topic. Spotted an error? Tell us and we will fix it.