Skip to content
Chalk Labs / AI SEO
How-to guide

How to Monitor AI Bot Activity in Server Logs

By Rahil Jain, founder of Chalk LabsUpdated How we research

The short answer

To see AI bots in your server logs, open your web server's access log and search the user-agent field for names such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Perplexity-User. Then check each visitor's IP address against the list the vendor publishes, because anyone can fake a user-agent name.

  • Analytics tools hide bot visits. Only server logs, or a CDN dashboard, show which AI bots fetched which pages.
  • Search for eight bot names across OpenAI, Anthropic and Perplexity, then group the hits by bot and by page.
  • A user-agent name proves nothing. Confirm the IP address against the vendor's published list before you trust a hit.
  • Look for 404s, blocked pages and bots that never visit your key pages. Those gaps explain missing citations.

What do you need, and how long does it take?

You need read access to your web server's access logs, or to the log view of your host or CDN. A first check takes about 30 minutes as a planning estimate. After that, a weekly review takes about 10.

You do not need to be an engineer, but you need someone who can download a log file or run a short command. If you use a managed host, ask support for "raw access logs for the last 30 days". If your site sits behind a CDN, the CDN often holds more complete logs than the origin server.

  • A log file covering at least 7 days, ideally 30.
  • A text search tool: the terminal commands below, a spreadsheet, or your host's log viewer.
  • A list of your 10 to 20 most important pages, so you can check whether the bots reached them.

Do it step by step

0 of 8 done0%

  1. Download the last 30 days of access logs from your server, host or CDN. Check the file has the user-agent field. Nginx and Apache combined format does.

    Tip: Log files are often split by day and compressed as .gz. Use zcat or gunzip to join them.

  2. Search the file for the eight bot names. Save the matching lines to a new file so you work with a small sample.

    Tip: Use a case-insensitive search. Vendors sometimes change the capitalisation.

  3. Group the matching lines by bot name. A table of eight rows and one number each is enough to start with.

  4. Pull the unique IPs for each bot. Test them against the vendor's published ranges. For Google, use the reverse and forward DNS check. Remove any hit that fails.

    Tip: Keep failed hits in a separate sheet. A burst of fake GPTBot visits is worth knowing about.

  5. For each verified bot, list the top 20 paths. Mark which of your 10 to 20 key pages appear and which do not.

  6. For each bot, count 200, 301, 403, 404 and 429 responses. Write down the 10 most-requested paths that returned a 404 or 403.

  7. Redirect the common 404 paths to the right pages. Unblock any verified bot that got a 403 or 429. Make sure key pages are linked from other pages and load their text without scripts.

  8. Run the same filters on the next 7 days. Record the numbers in one sheet. Check that the fixes moved the 404 share and that previously missing pages now appear.

    Tip: Wait at least a week before judging a fix. Crawl visits are uneven.

Your ticks are saved in this browser only.

Copy-ready templates

Count AI bot hits and their status codes (Nginx or Apache combined log)

bash
LOG=/var/log/nginx/access.log
BOTS='OAI-SearchBot|GPTBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User'

# 1. Hits per bot
grep -ioE "$BOTS" "$LOG" | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn

# 2. Status codes for all AI bot hits (field 9 in combined format)
grep -iE "$BOTS" "$LOG" | awk '{print $9}' | sort | uniq -c | sort -rn

# 3. Top 20 paths requested by AI bots (field 7)
grep -iE "$BOTS" "$LOG" | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# 4. Paths that returned 404 to AI bots
grep -iE "$BOTS" "$LOG" | awk '$9==404 {print $7}' | sort | uniq -c | sort -rn | head -20

# 5. Unique IPs per bot, to verify next
grep -i 'PerplexityBot' "$LOG" | awk '{print $1}' | sort -u > perplexitybot-ips.txt

Field numbers assume the standard combined format. If your host uses a custom format, adjust $7 and $9. For .gz files, replace grep ... "$LOG" with zcat file.gz | grep ...

Python script: Check IPs against a vendor's published range file

text
# usage: python3 check_ips.py <range-json-url> < ips.txt
# example: python3 check_ips.py https://openai.com/searchbot.json < searchbot-ips.txt
import ipaddress, json, sys, urllib.request

req = urllib.request.Request(sys.argv[1], headers={"User-Agent": "Mozilla/5.0 (log-check)"})
data = json.load(urllib.request.urlopen(req))
nets = []
for p in data.get("prefixes", []):
    for key in ("ipv4Prefix", "ipv6Prefix"):
        if key in p:
            nets.append(ipaddress.ip_network(p[key], strict=False))

for line in sys.stdin:
    line = line.strip()
    if not line:
        continue
    ip = ipaddress.ip_address(line)
    print(ip, "MATCH" if any(ip in n for n in nets) else "NOT IN LIST")

Save as check_ips.py. The range files use a prefixes array with ipv4Prefix entries when checked on 20 October 2026. Open the file once to confirm the layout, since vendors can change it. Use one URL per bot: the OpenAI, Perplexity and Anthropic files named on this page.

Verify a Googlebot IP with reverse and forward DNS

bash
# Step 1: reverse lookup. The name should end in googlebot.com, google.com or googleusercontent.com
host 66.249.66.1

# Step 2: forward lookup on the name returned in step 1
host crawl-66-249-66-1.googlebot.com

# Step 3: the forward lookup must return the original IP address

66.249.66.1 is one of the example addresses in Google's own guide. Replace it with an IP from your logs, and use the hostname step 1 returns.

User-agent filter for log tools and CDN rules

regex
(OAI-SearchBot|GPTBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User)

Paste into the user-agent filter of any log viewer that accepts regular expressions. Make the match case-insensitive.

Why do server logs show AI bots when analytics does not?

Server logs record every request the server receives, including requests from bots. Analytics tools do not. Google says traffic from known bots and spiders is automatically excluded from Google Analytics, and you cannot switch that off 8.

Many AI crawlers also never run your page scripts. A Vercel study of its network, published in December 2024, found that none of the major AI crawlers it measured render JavaScript 7. That means a script-based tracker would not fire for them even if bots were not filtered. The study is almost two years old, so treat it as a pattern to test on your own logs, not a fixed fact.

The result: if you want to know whether ChatGPT, Claude or Perplexity has fetched a page, the access log is the first place to look. For human visits that arrive from AI answers, use GA4 tracking instead.

Which AI bot names should you search for?

Search for the eight names in the table. They come from the vendors' own documentation. Each vendor runs several bots with different jobs, so a hit from one does not tell you about the others 123.

The most useful split is crawler versus user-triggered fetcher. A crawler visits on its own schedule. A user-triggered fetcher visits because a person asked a question and the assistant needed your page. OpenAI says ChatGPT-User does not crawl automatically, and that robots.txt rules may not apply to it 1. Perplexity says Perplexity-User generally ignores robots.txt because a user requested the fetch 3.

AI bot names to search for in your logs
Search forRun byWhat it does, per the vendorrobots.txt per vendor
OAI-SearchBotOpenAISurfaces sites in ChatGPT search featuresYes, it is the search opt-out 1
GPTBotOpenAICrawls content that may train OpenAI modelsYes, disallowing signals no training use 1
ChatGPT-UserOpenAIFetches a page when a user asks ChatGPTMay not apply 1
ClaudeBotAnthropicCollects web content that could train ClaudeHonoured 2
Claude-SearchBotAnthropicIndexes content to improve search resultsHonoured 2
Claude-UserAnthropicRetrieves pages when a Claude user's question needs itNot separately stated on the page 2
PerplexityBotPerplexitySurfaces and links sites in Perplexity resultsAllow it if you want to appear 3
Perplexity-UserPerplexityFetches pages for a user's questionGenerally ignores robots.txt 3

What does one log line look like?

Nginx writes the "combined" log format by default, and many Apache setups use it too, though some Apache installs default to the shorter "common" format, which has no user-agent field. In Nginx the combined format holds the client IP, time, request line, status code, bytes sent, referrer and user agent, in that order 6. The bot name sits in the last quoted field.

The example below is invented to show the layout. Its IP address comes from a range reserved for documentation, so it will not match any real list.

One combined-format log line (illustrative)
  • 203.0.113.10 - - [20/Oct/2026:09:14:03 +0000] "GET /pricing HTTP/1.1" 200 18342 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
  • Field 1: client IP. Field 4-5: date and time. Field 6-8: request line. Field 9: status. Last quoted field: user agent.

How do you tell a real AI bot from a fake one?

Check the IP address, not the name. Any script can send "GPTBot" in its user-agent header, so the name alone proves nothing. The vendor-published IP lists are the check.

OpenAI publishes IP range files for each bot, such as searchbot.json and gptbot.json 1. Anthropic publishes a crawler IP list at claude.com/crawling/bots.json 2. Perplexity publishes perplexitybot.json and perplexity-user.json 3. Download the file, then test each IP from your logs against its ranges with the script in the templates.

Google uses a different method. Run a reverse DNS lookup on the IP, confirm the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP 4. Google also publishes JSON range files such as common-crawlers.json for bulk checks 4.

One warning. Anthropic says blocking by IP address may not reliably work as an opt-out, because it stops the bot reading robots.txt. Use IP lists to verify visitors, and robots.txt to set rules 2.

What patterns in the logs should worry you?

The first pattern is a high share of 404 responses. In the Vercel study, 34.82% of ChatGPT's fetches and 34.16% of Claude's hit 404 pages, against 8.22% for Googlebot 7. Those figures are from late 2024, so measure your own. If a bot keeps asking for URLs that do not exist, use 301 redirects for hallucinated URLs.

The second pattern is a bot that reaches only your home page. If PerplexityBot fetched your robots.txt and home page but never your service or pricing pages, check internal links and whether those pages load without scripts. See server-side rendering for AI.

The third pattern is a missing bot. If OAI-SearchBot never appears over 30 days, check robots.txt and any firewall or CDN bot rule. Our guide to allowing AI crawlers in robots.txt lists the lines to check. OpenAI says robots.txt changes can take about 24 hours to take effect for search 1.

A fourth pattern is a 403 or 429 response to a verified bot. That means your firewall is blocking the visitor you want, whatever robots.txt says.

Do you need a dashboard, or is a spreadsheet enough?

A spreadsheet is enough for most small sites. Export one row per bot per week: hits, share of 404s, and number of distinct pages fetched. Add a column for hits to your 10 to 20 key pages.

If you use Cloudflare, its AI Crawl Control feature shows which AI services access your site and lets you set allow or block rules per crawler. Cloudflare says it is available on all plans 9. Other CDNs and log tools offer similar filters. Pick one that can group by user agent and by status code, and you will have what you need.

Whatever you pick, compare the same window each week. A single day of logs says little, because crawl visits come in bursts.

What mistakes do people make?

The most common mistake is counting every hit as a real bot. Always verify IPs before you report numbers to anyone.

The second mistake is confusing training bots with search bots. Blocking GPTBot does not remove you from ChatGPT search, because OpenAI says each robots.txt setting is independent 1. Our page on blocking AI training but allowing AI search covers the rules.

The third mistake is reading bot visits as rankings or citations. A fetch means the bot reached the page. It does not mean the page was used in an answer. To check citations, run your questions in the assistants themselves, as in our AI visibility audit guide.

When is log monitoring not worth the effort?

Skip it if you cannot get logs at all, for example on some website builders. Use the AI Crawler Checker to test what the bots are allowed to read, and track mentions by hand instead.

Skip the weekly routine if your site has under a few hundred visits a month and you have not yet fixed the basics. Do the crawl check first, then come back to logs once the pages are live and linked.

Frequently asked questions

Why can't I see GPTBot or ClaudeBot in Google Analytics?

Google Analytics automatically excludes traffic from known bots and spiders, and you cannot turn that off 8. Many AI crawlers also do not run page scripts 7. Server logs and CDN dashboards record the raw requests, so use those instead.

Does a GPTBot hit mean ChatGPT will cite my page?

No. OpenAI says GPTBot crawls content that may be used for training, while OAI-SearchBot surfaces sites in ChatGPT search 1. Both visits only show a fetch. Whether a page is cited depends on the answer, so test your questions in ChatGPT directly.

Why do I see ChatGPT-User or Perplexity-User even though I blocked them?

OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person triggered the visit 1. Perplexity says Perplexity-User generally ignores robots.txt for the same reason 3. If you need a hard block, use a firewall rule, and accept the loss of visibility.

How often do the vendors update their IP lists?

The pages we read give no fixed update schedule for the IP files. Re-download the files before each review rather than caching them for months, and treat a list that is much older than your logs with care.

What if my host does not give me logs?

Ask whether a CDN in front of the site can log requests. If neither is possible, run the AI Crawler Checker to test access, and track citations by hand. Some managed website builders simply do not expose logs.

About the author

Rahil Jain

Rahil Jain is an AI SEO (answer engine and generative engine optimisation) consultant and co-founder of Chalk Labs.

LinkedIn · Method and credentials

Sources

  1. Overview of OpenAI Crawlers (Bots) · OpenAIOpened October 20, 2026. Supports: OAI-SearchBot, GPTBot, ChatGPT-User purposes; robots.txt behaviour; IP range file names; about 24 hours for search opt-out changes.
  2. Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic (Claude Help Center)Opened October 20, 2026. Supports: ClaudeBot, Claude-User and Claude-SearchBot roles; robots.txt honoured; published IP list; warning about blocking by IP. Page last updated 7 April 2026.
  3. Perplexity Crawlers · PerplexityOpened October 20, 2026. Supports: PerplexityBot and Perplexity-User roles, user agents, robots.txt behaviour and IP JSON files.
  4. Verifying Googlebot and other Google crawlers · Google Search CentralOpened October 20, 2026. Supports: Reverse and forward DNS verification and the published JSON IP range files. Page last updated 2026-03-20.
  5. Google's common crawlers · Google Search CentralOpened October 20, 2026. Supports: Google-Extended has no separate user agent string; Googlebot token. Page last updated 2026-10-15.
  6. Module ngx_http_log_module · nginx.orgOpened October 20, 2026. Supports: The predefined combined log format and its field order.
  7. The rise of the AI crawler · VercelOpened October 20, 2026. Supports: Published 17 December 2024: no major AI crawler measured rendered JavaScript; 404 shares of 34.82% (ChatGPT), 34.16% (Claude) and 8.22% (Googlebot).
  8. Exclude known bots and spiders traffic (Google Analytics Help) · GoogleOpened October 20, 2026. Supports: Traffic from known bots and spiders is automatically excluded and cannot be switched off.
  9. AI Crawl Control · Cloudflare DocsOpened October 20, 2026. Supports: Dashboard of AI crawler activity, per-crawler allow or block rules, available on all plans.
How this page was made. Built on Rahil Jain's AI SEO method and drafted with AI research assistance. Every statistic links to the page we read it on (opened October 20, 2026 or nearby). Published October 20, 2026; last updated October 20, 2026. Chalk Labs sells AI SEO services, so we have a commercial interest in this topic. Spotted an error? Tell us and we will fix it.