How-to guide

How to Check If ChatGPT and Other AI Bots Can Crawl Your Website

By Rahil Jain, founder of Chalk LabsUpdated October 20, 2026How we research

The short answer

To check if ChatGPT can crawl your website, test four gates: your robots.txt rules for OAI-SearchBot (OpenAI's search bot), any firewall or CDN block, the page's status code and noindex tags, and whether your text sits in the raw HTML. Then confirm with server logs, which record real visits. A free crawler checker covers two of the four in minutes.

OAI-SearchBot is the bot OpenAI ties to ChatGPT search: opted-out sites are not shown. GPTBot is for training. ChatGPT-User fetches pages on request.
Four gates can stop a bot: robots.txt, a firewall or CDN, the page response and tags, and JavaScript-only text.
Server logs, not a curl test with a bot's name, show which bots really visited.
Most AI crawlers did not run JavaScript in the last public test we can cite, from late 2024. Check your raw HTML.
Passing every check does not mean ChatGPT will cite you. It removes one reason you could be missing.

Where can an AI bot be stopped before it reads your page?

Four things can stop an AI crawler, which is a program that fetches web pages for an AI company. Start with gate 1, the quickest to check. The steps below run the script tests first and the dashboard check after.

Server logs make a fifth step: the file where your server records every request, and the best evidence of what a bot did.

Passing every gate does not mean ChatGPT will cite you. Google says the same for its own AI features: a page needs to be indexed and eligible to show with a snippet, with no extra technical requirements ¹⁴.

Gate 1, robots.txt: a text file at /robots.txt that tells bots which paths they may fetch.
Gate 2, firewall or CDN: a security layer, often called a WAF (web application firewall), that can refuse a bot before your site answers, by its name (its user agent) or its address. A CDN serves your site from servers near each visitor.
Gate 3, page response and tags: the status code your server returns, such as 200 for success or 403 for refused, plus noindex tags, which tell search engines to leave a page out of results.
Gate 4, raw HTML: the text in the page before any JavaScript runs.

0 of 8 done0%

Open your robots.txt and make sure it loads
Open yourdomain.com/robots.txt in a browser. Then run curl -sL -o /dev/null -w '%{http_code}' https://yourdomain.com/robots.txt and expect 200, because a 404 or a 5xx changes what crawlers assume ⁴. Repeat for every subdomain with public pages, because Anthropic says block rules go in the top-level file once per subdomain ².
Read what robots.txt says for each bot
Run the Python robots check from the templates, or use the free AI Crawler Checker. Confirm OAI-SearchBot shows ALLOWED for your key pages, then check ChatGPT-User, PerplexityBot, Claude-SearchBot and Googlebot. Fix mistakes with the explicit-allow block.
Tip: Python's parser ignores * and $ inside paths, so use the checker for edge cases.
Test a key page with each bot's user agent
Run the user agent test on your home, pricing or service page. Expect 200 for every line. A 403, 429 or 503 for one bot points to a firewall, rate limit or server problem.
Tip: A pass here is a hint, not proof. See the warning in the firewall section ¹⁹.
Read the raw HTML of your key page
Run the raw HTML script with a sentence from the page. You want “Phrase in text outside scripts: True”, a real title and H1, and no noindex. If the phrase is missing, confirm with JavaScript disabled in Chrome ¹⁵.
Tip: A phrase found only under “anywhere in HTML” sits in a script tag, which some bots skip.
Check your CDN or firewall dashboard
In Cloudflare, open AI Crawl Control and look for any crawler set to Block. Then read the AI bot policies in Security Settings and filter security events by bot name ⁵⁷⁹. Elsewhere, check bot protection and rules that block by user agent, country or request rate.
Tip: Write down every setting before you change it. You may see the older Block AI bots setting instead ⁵.
Search your server logs for the bot names
Run the log commands from the templates. Count hits for each bot and read the status codes for OAI-SearchBot. A 200 on key pages is good, and any 403 or 429 confirms a block.
Verify the visiting addresses
Run the verification script on your log. It compares each visit with the ranges OpenAI, Perplexity and Anthropic publish ¹²³. Treat hits outside the ranges as unverified: old log lines can fall outside today's ranges, so check any outlier by hand before you block it.
Tip: Behind a CDN, make sure the log shows the real visitor address.
Fix one thing, wait a day, then record the result
Change one setting at a time, then wait about 24 hours for OpenAI's search systems to adjust ¹. Repeat steps 1 to 4, check for new log hits, and note the result with the date. CDN defaults change, so repeat monthly ⁵.
Tip: Keep a copy of the old robots.txt and firewall rule so you can roll back.

Your ticks are saved in this browser only.

Explicit-allow robots.txt block for AI search and user bots

text

# Rules for everyone else (keep your own rules here)
User-agent: *
Disallow: /admin/

# AI search and user bots: allowed everywhere except /admin/.
# A bot with its own group ignores the group above,
# so shared rules must be repeated here.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /admin/
Allow: /

Replace /admin/ with your own private paths. GPTBot and ClaudeBot are left out on purpose, so they follow the * group. Test the file with the Python robots check before you upload it.

Python 3: what does robots.txt say about each AI bot?

text

import sys
from urllib.error import HTTPError
from urllib.request import Request, urlopen
from urllib.robotparser import RobotFileParser

site = sys.argv[1].rstrip("/") if len(sys.argv) > 1 else "https://example.com"
page = sys.argv[2] if len(sys.argv) > 2 else site + "/"
BOTS = ["OAI-SearchBot", "ChatGPT-User", "GPTBot", "Claude-SearchBot",
        "Claude-User", "ClaudeBot", "PerplexityBot", "Perplexity-User",
        "Googlebot", "Bingbot"]

req = Request(site + "/robots.txt", headers={"User-Agent": "robots-check/1.0"})
status, body = None, ""
try:
    with urlopen(req, timeout=20) as r:
        status, body = r.status, r.read().decode("utf-8", "replace")
except HTTPError as e:
    status = e.code
except Exception as e:
    print("Could not reach the site:", e)

print("robots.txt status:", status)
if status == 200:
    rp = RobotFileParser()
    rp.parse(body.splitlines())
    for bot in BOTS:
        verdict = "ALLOWED" if rp.can_fetch(bot, page) else "BLOCKED"
        print(f"{bot:16} {verdict}  {page}")
elif status and 400 <= status < 500:
    print("4xx: crawlers may treat this as no rules. If this is a 403, your firewall may have refused this script; compare with ua-test.sh on /robots.txt.")
elif status:
    print("5xx: crawlers may treat the whole site as off limits. Fix this first.")

Save as robots_check.py and run python3 robots_check.py https://yourdomain.com https://yourdomain.com/pricing. It prints ALLOWED or BLOCKED for each bot. Python's parser is a rough check: it reads rules in file order and ignores * and $ inside paths. ChatGPT-User and Perplexity-User may ignore robots.txt anyway.

Bash: test a page with each bot's user agent

bash

#!/usr/bin/env bash
# Usage: bash ua-test.sh https://yourdomain.com/your-key-page
URL="${1:?Usage: bash ua-test.sh https://example.com/page}"

try () {
  code=$(curl -s -o /dev/null -L -m 20 -w "%{http_code}" -A "$2" "$URL")
  printf "%-16s %s\n" "$1" "$code"
}

try "Browser"         "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
try "OAI-SearchBot"   "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
try "GPTBot"          "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
try "ChatGPT-User"    "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
try "PerplexityBot"   "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
try "Perplexity-User" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)"
try "Claude-SearchBot" "Mozilla/5.0 (compatible; Claude-SearchBot/1.0)"
try "ClaudeBot"       "Mozilla/5.0 (compatible; ClaudeBot/1.0)"

Save as ua-test.sh and run it. The OpenAI and Perplexity strings are copied from their documentation. Anthropic's help page gives bot names, not full strings, so the two Claude lines use short strings. Version numbers change, so a firewall rule should match the bot name.

Python 3: is your text in the raw HTML?

text

# Usage: python3 raw_html_check.py https://example.com/page "a sentence from the page"
import re, sys
from html import unescape
from urllib.error import HTTPError
from urllib.request import Request, urlopen

url, phrase = sys.argv[1], sys.argv[2]
req = Request(url, headers={"User-Agent": "Mozilla/5.0 (compatible; raw-html-check)"})
try:
    with urlopen(req, timeout=20) as r:
        status, headers, html = r.status, r.headers, r.read().decode("utf-8", "replace")
except HTTPError as e:
    sys.exit(f"The server answered {e.code}. A bot may get the same answer.")

no_code = re.sub(r"<(script|style)\b.*?</\1>", " ", html, flags=re.S | re.I)
text = re.sub(r"\s+", " ", unescape(re.sub(r"<[^>]+>", " ", no_code)))

def first(pattern):
    m = re.search(pattern, html, flags=re.I | re.S)
    return unescape(re.sub(r"<[^>]+>", "", m.group(1))).strip() if m else "(none)"

print("Status:", status, "| HTML bytes:", len(html), "| words outside scripts:", len(text.split()))
print("Phrase in text outside scripts:", phrase.lower() in text.lower())
print("Phrase anywhere in HTML (scripts count):", phrase.lower() in html.lower())
print("Title:", first(r"<title[^>]*>(.*?)</title>"))
print("H1:", first(r"<h1[^>]*>(.*?)</h1>"))
print("Robots meta:", first(r"<meta[^>]+name=[\"']?robots[\"']?[^>]*content=[\"']?([^\"'>]*)"))
print("X-Robots-Tag header:", headers.get("X-Robots-Tag", "(none)"))

Save as raw_html_check.py. Pick a sentence that only appears on that page. A very low word count on a long page suggests the content loads with JavaScript. Text inside tabs or CSS-hidden text still counts here, even though some bots may skip it. The robots meta check misses tags that put content before name, so also search the page source for “noindex”. The script sends a plain browser-style request, so a firewall that blocks it will also show up here.

Bash: count AI bot hits in an access log

bash

LOG=access.log   # your log path. For rotated .gz files, use zgrep instead of grep.

# 1. How many requests did each AI bot make?
for b in OAI-SearchBot GPTBot ChatGPT-User Claude-SearchBot Claude-User ClaudeBot PerplexityBot Perplexity-User; do
  printf "%-17s %s\n" "$b" "$(grep -c "$b" "$LOG")"
done

# 2. What status codes did one bot get? Field 9 in the standard log format.
grep 'OAI-SearchBot' "$LOG" | awk '{print $9}' | sort | uniq -c | sort -rn

# 3. Which pages did it ask for, and what happened?
grep 'OAI-SearchBot' "$LOG" | awk '{print $7, $9}' | sort | uniq -c | sort -rn | head -20

# 4. Has OAI-SearchBot read your robots.txt? OpenAI adds a marker to its user agent for that.
grep 'OAI-SearchBot' "$LOG" | grep 'robots.txt;' | tail -3

Works with the standard nginx and Apache combined log format, where field 7 is the path and field 9 is the status code. Other formats need different field numbers. The match is case-sensitive on purpose: the bot names are written exactly like this, and a lowercase match would also hit the bot's URL.

Python 3: check bot IP addresses against published ranges

text

#!/usr/bin/env python3
"""Usage: python3 verify_bots.py access.log
Counts AI-bot hits in a web server log and checks each source IP
against the vendor's published IP ranges."""
import ipaddress, json, sys
from collections import Counter
from urllib.request import Request, urlopen

FILES = {
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "GPTBot": "https://openai.com/gptbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "PerplexityBot": "https://www.perplexity.com/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.com/perplexity-user.json",
    "Claude-SearchBot": "https://claude.com/crawling/bots.json",
    "Claude-User": "https://claude.com/crawling/bots.json",
    "ClaudeBot": "https://claude.com/crawling/bots.json",
}

def load(url):
    req = Request(url, headers={"User-Agent": "bot-verify/1.0"})
    data = json.load(urlopen(req, timeout=20))
    cidrs = (p.get("ipv4Prefix") or p.get("ipv6Prefix") for p in data.get("prefixes", []))
    return [ipaddress.ip_network(c, strict=False) for c in cidrs if c]

cache = {}
nets = {}
for bot, url in FILES.items():
    if url not in cache:
        cache[url] = load(url)
    nets[bot] = cache[url]

verified, claimed = Counter(), Counter()
fakes = Counter()
with open(sys.argv[1], errors="replace") as log:
    for line in log:
        low = line.lower()
        bot = next((b for b in FILES if b.lower() in low), None)
        if not bot:
            continue
        ip_text = line.split(" ", 1)[0]
        try:
            ip = ipaddress.ip_address(ip_text)
        except ValueError:
            continue
        if any(ip in n for n in nets[bot]):
            verified[bot] += 1
        else:
            claimed[bot] += 1
            fakes[(bot, ip_text)] += 1

for bot in FILES:
    if verified[bot] or claimed[bot]:
        print(f"{bot:17} in published ranges: {verified[bot]:5}   not in ranges: {claimed[bot]:5}")
for (bot, ip), n in fakes.most_common(10):
    print(f"  not in {bot} ranges: {ip} ({n} hits)")

Save as verify_bots.py. It downloads the current range files on every run, because the ranges change. It assumes the visitor address is the first field of each log line, which is not true behind some proxies and CDNs. A hit outside the ranges is unverified, not proof of a fake, because the script compares old log lines with today's ranges. The Claude lines share one Anthropic list. For Googlebot, use reverse DNS or Google's range files instead.

Results log to copy into a sheet

markdown

| Check | Command or screen | Expected | Result | Date |
|---|---|---|---|---|
| robots.txt loads | curl status on /robots.txt | 200 | | |
| OAI-SearchBot allowed | robots check script | ALLOWED | | |
| Bot user agents get 200 | ua-test.sh | 200 for all | | |
| Text in raw HTML | raw_html_check.py | Phrase in text outside scripts: True | | |
| No noindex | raw_html_check.py | Robots meta and header: (none) | | |
| CDN shows no AI block | AI Crawl Control or security events | No blocks for search bots | | |
| Bot hits in logs | log commands | At least one 200 for OAI-SearchBot | | |
| Visits from real ranges | verify_bots.py | Any hit outside the ranges checked by hand | | |

Re-run it monthly, and after any CDN, hosting or site-platform change. Save each month's copy so you can see when something changed.

What do you need, and how long does it take?

A first pass takes under an hour on a small site (our estimate, not a measured figure). We ran the scripts below on 20 October 2026 with Python 3.11 and bash on macOS.

Not technical? Run the free AI Crawler Checker yourself and hand the bot-name test, log search and address check to a developer or your host.

A browser, plus a terminal with curl (a tool that fetches a web address and shows the raw reply) and Python 3, which is a free install. Macs and Linux include curl. On Windows, run the bash scripts in WSL or Git Bash.
Login access to your CDN or firewall dashboard, and your server's access logs. Some hosts keep logs only briefly, so pull them early.

Why check crawl access before you write more content?

A blocked bot cannot read your best page, so no edit to the page can help. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links ¹.

Blocks can be accidental. On 1 July 2025, Cloudflare announced it was changing its default to block AI crawlers unless they pay creators for content ⁶. Cloudflare's documentation, last updated on 1 July 2026, said that on 15 September 2026 it would set updated defaults for new domains: Training and Agent bots blocked on pages that display ads, Search allowed ⁵. That date has passed and the page still uses future tense, so we cannot confirm what your zone received. It also marks the older Block AI bots setting as deprecating that day, so you may see either screen.

The 2025 post does not say how far the change reaches into existing sites, so read your own settings ⁶.

Which AI bots should you look for?

Each AI company runs several bots, each with its own name, called a user agent token. Use that name in robots.txt, firewall rules and log searches. OpenAI runs three bots that matter for most sites. OAI-SearchBot is the one tied to ChatGPT search: sites that opt out of it are not shown in ChatGPT search answers ¹.

OpenAI also lists OAI-AdsBot, which checks the safety of pages submitted as ads on ChatGPT ¹. Full user agent strings carry version numbers that change, so search by bot name, not the whole string ¹. For more detail, read what AI crawlers are and GPTBot vs OAI-SearchBot vs ChatGPT-User.

AI bot names to check in robots.txt and in your logs
Bot name	Company	What it does	Does robots.txt control it?
OAI-SearchBot	OpenAI	Finds pages for ChatGPT search results ¹.	Yes. OpenAI recommends allowing it ¹.
ChatGPT-User	OpenAI	Opens a page when a person asks ChatGPT to ¹.	Maybe not. OpenAI says robots.txt may not apply ¹.
GPTBot	OpenAI	Collects content that may train models ¹.	Yes. Disallowing it signals no training use ¹.
Claude-SearchBot	Anthropic	Navigates the web to improve search quality ².	Yes. Blocking it may reduce your search visibility ².
Claude-User	Anthropic	Fetches a page when a Claude user asks ².	Yes. Blocking it may reduce visibility in user-directed search ².
ClaudeBot	Anthropic	Collects content that could contribute to training ².	Yes. Anthropic says its bots honor robots.txt ².
PerplexityBot	Perplexity	Surfaces and links sites in Perplexity results ³.	Yes. Perplexity recommends allowing it ³.
Perplexity-User	Perplexity	Fetches a page when a user asks ³.	It generally ignores robots.txt, since a user asked ³.
Googlebot	Google	Crawls for Google Search. Pages must be indexed and snippet-eligible to appear in AI Overviews and AI Mode ¹⁴.	Yes. Google's crawlers obey robots.txt ¹³.
Google-Extended	Google	A robots.txt token for Gemini training and grounding. It has no user agent, so it never appears in logs ¹³.	Yes. It does not affect Google Search inclusion ¹³.

How does a bot read your robots.txt file?

A bot obeys the group of rules that names it. If no group names it, the bot uses the * group, which means everyone else ⁴.

This has a trap. Once you add a group for OAI-SearchBot, the * rules stop applying to that bot. If you blocked /admin/ for everyone, repeat that rule inside the OAI-SearchBot group ⁴.

When an allow rule and a disallow rule both match, the longer path wins. If they tie, allow wins ⁴.

The file itself must load. A 4xx error, such as 404, lets crawlers fetch anything. A 5xx server error means crawlers must treat the whole site as off limits ⁴.

ChatGPT-User and Perplexity-User act because a person asked, and both companies say robots.txt may not apply to them ¹³. The standard itself says robots.txt is not access control ⁴.

Worked example: one file, three different answers

User-agent: *
Disallow: /admin/
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
OAI-SearchBot may fetch /pricing and /admin/. Its own group replaces the * group.
GPTBot may fetch nothing.
PerplexityBot has no group, so it follows *. It may fetch /pricing but not /admin/.

How can you tell if Cloudflare or a firewall is blocking AI bots?

A firewall block does not show in robots.txt, so your file can say yes while the firewall says no. The AI Crawler Checker cannot see firewall rules either, because it reads your site from outside.

In Cloudflare, start with AI Crawl Control. It lists each AI crawler with its request count and an Allow or Block setting, and charts status codes, including 403 blocked responses ⁷.

Next, open Security Settings and find the AI bot policies. Cloudflare offers three presets, Search, Agent and Training, and each can block on all pages, block on pages with ads, or allow ⁵. Agent covers real-time activity for a person, such as chat fetch bots ⁵. ChatGPT-User looks like that kind of bot, so check your Agent setting.

Bot Fight Mode is a separate free setting that challenges suspected bots. WAF custom rules cannot skip it, but Super Bot Fight Mode can ⁸.

To see real blocks, open security events and filter by user agent. Cloudflare names the Analytics page and its Events tab, and warns that sampled logs may not show every event ⁹.

Other firewalls work alike. Perplexity says its bots may need an allow rule that matches both user agent and published IP ranges, and it gives Cloudflare and AWS WAF steps ³.

Do AI crawlers run JavaScript?

Most did not, in the last public test we can cite. Vercel's study, published in December 2024, found that OpenAI's crawlers (GPTBot, OAI-SearchBot and ChatGPT-User), Anthropic's ClaudeBot and PerplexityBot did not render JavaScript ¹⁰. Google's Gemini, which uses Googlebot's infrastructure, and Apple's crawler did ¹⁰.

That result is almost two years old. The OpenAI, Anthropic and Perplexity crawler pages we read on 20 October 2026 do not say whether their bots run JavaScript ¹²³. Treat the study as a warning, not a current rule, and test your own pages.

Google renders pages from a queue, yet still recommends server-side or pre-rendering because not all bots run JavaScript ¹¹. Microsoft adds that AI systems may not render hidden content, such as text inside tabs or expandable menus ¹². Put the answer in plain visible text.

Test it in two minutes. In Chrome, open DevTools, press Control+Shift+P (Command+Shift+P on a Mac), type javascript and choose Disable JavaScript ¹⁵. If your text vanishes, a bot that skips scripts will not see it.

If text is missing, read how to make your site server-side rendered.

How do you check server logs for GPTBot visits?

Search your access log for the bot names, count the hits, then check that the visiting IP addresses belong to the company. An access log records each visitor's address, page, status code and user agent.

OpenAI may add a robots.txt marker to the user agent when it fetches your robots.txt file, which shows the bot is reading your rules ¹. If you use Cloudflare, AI Crawl Control shows crawler activity without log files ⁷.

Anyone can type a bot's name into their user agent, so verify the address. OpenAI publishes IP ranges for OAI-SearchBot, GPTBot and ChatGPT-User as JSON files ¹. Perplexity does the same for its two bots ³, and Anthropic publishes one list ². For Googlebot, Google says to use a reverse DNS lookup, which checks the hostname behind an address, or its range files ¹⁶.

Do not expect large numbers. ChatGPT-User appears only when a person asks ChatGPT to open your page. If you allow both OAI-SearchBot and GPTBot, OpenAI says it may use one crawl for both ¹. Google-Extended never appears, because it has no user agent string ¹³.

Read the status codes too. A 200 means the bot got the page, a 403 or 429 means something refused it, and a 404 means it asked for a page that does not exist.

Dead URLs deserve attention. In Vercel's December 2024 study, 34.82% of ChatGPT crawler fetches and 34.16% of Claude crawler fetches hit 404 pages, against 8.22% for Googlebot ¹⁰. Vercel calls the cause unclear and points to requests for outdated /static/ assets ¹⁰. Models can also invent URLs: see what hallucinated URLs are. For a longer walkthrough, read how to monitor AI bot activity in server logs.

What mistakes waste the most time, and which gate does each break?

Match what you see to the gate that broke.

Common crawl blocks and their fixes
What you see	Gate	How to confirm it	Fix
A staging rule went live: `Disallow: /` under `User-agent: *`.	1	Open /robots.txt and look for a lone slash.	Narrow or remove the rule.
robots.txt returns a 5xx error.	1	Run the status command in step 1.	Fix the server ⁴.
A named bot group dropped your shared rules.	1	Compare each named group with the `*` group.	Repeat shared rules in every group ⁴.
A Cloudflare preset blocks Training or Agent bots.	2	AI Crawl Control shows Block, or 403s for the bot.	Change the preset or set a per-crawler Allow ⁵⁷.
Bot Fight Mode challenges the bot.	2	Security events show challenges for the user agent.	Use Super Bot Fight Mode exceptions or an IP Access rule that matches first ⁸.
A CAPTCHA or login sits in front of the page.	3	Request the page with curl and read the reply.	Serve public pages without a challenge. Anthropic's bots will not bypass CAPTCHAs ².
`noindex` in a meta tag or `X-Robots-Tag` header.	3	Run the raw HTML script.	Remove it where you want to be found. Google lists noindex among its Search controls ¹⁴. OpenAI does not say how its bots treat it, so count it as a ChatGPT risk, not a rule ¹.
Key text appears only after JavaScript runs, or sits in tabs or images.	4	Disable JavaScript in DevTools.	Render on the server, and use plain visible text ¹⁰¹¹¹².

What did an AI assistant search for when we asked this question?

On 20 October 2026 we gave a Claude agent with web search the question “how to check if ChatGPT can crawl my website” and told it to run 6 to 10 searches. It ran six, listed in the panel on this page.

We set that range, so the count says nothing about how many searches Claude runs on its own. This was one run on one engine: a sample, not a rule.

How do you know your fix worked?

Your fix worked when four things line up. Your robots check shows ALLOWED for OAI-SearchBot. Your key page returns 200 for every bot test. Your text is in the raw HTML, and an address from the company's published ranges shows in your logs.

Wait a day before you judge. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt update ¹, and Perplexity says up to 24 hours ³.

Then use outside reports. Bing Webmaster Tools announced an AI Performance report in public preview on 10 February 2026. It shows how often your content is cited across Microsoft Copilot, Bing's AI summaries and select partner integrations ¹⁸. Google's Generative AI performance report in Search Console shows impressions in AI Overviews and AI Mode. Google says it reached all websites worldwide by 31 August 2026 ²¹.

Neither report names ChatGPT as a source it covers. For ChatGPT, your logs and your own test prompts are the evidence we can point to. How to test whether your changes worked shows a method.

When is a crawl check not worth your time?

Skip it if you blocked AI bots on purpose, but make sure the block separates training bots from search bots.

Pause if you cannot change the settings. Shopify, for example, lets merchants edit robots.txt only through a template file and calls that an unsupported customisation ¹⁷. Ask your platform first.

Once the four gates pass, move on to content. Read how to allow AI crawlers in robots.txt if a fix is needed, or how to write content AI engines quote if access is clean.

What an AI assistant searches when it answers this

We gave the question below to AI search tools and recorded the web searches they ran behind the scenes. This is called query fan-out. Each phrase is a page you could publish to be found and cited.

The prompt

“how to check if ChatGPT can crawl my website”

The searches it ran

check if ChatGPT can crawl my website
GPTBot OAI-SearchBot robots.txt user agents
test if AI crawlers blocked by Cloudflare
AI crawler checker tool free
do AI crawlers execute JavaScript rendering
check server logs for GPTBot visits

Observed October 20, 2026 using Claude web search. Claude agents were told to run 6 to 10 searches, so that count is our setting, not Claude's own choice. Assistants differ and change often, so treat this as a sample, not a rule. See how we collect it.

Frequently asked questions

Can ChatGPT crawl my website if I have never changed anything?

It depends on settings you may not know about. If nothing in your robots.txt, firewall or CDN refuses OAI-SearchBot, nothing stops it fetching public pages. A CDN default or a leftover Disallow rule can block it silently, so test.

Does blocking GPTBot stop ChatGPT from finding my site?

No. OpenAI treats each robots.txt setting as independent, so you can disallow GPTBot (training) and still allow OAI-SearchBot (search) ¹. Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers ¹. Check which name your rule uses.

Why did ChatGPT-User visit when my robots.txt blocks it?

ChatGPT-User acts when a person asks ChatGPT to open a page, and OpenAI says robots.txt rules may not apply to those visits ¹. Perplexity says the same about Perplexity-User ³. Put private pages behind a login or server rule, because robots.txt is a request, not a lock ⁴.

Is there a free tool that checks AI crawler access?

Yes. The free AI Crawler Checker reads your robots.txt for named AI bots and checks whether your homepage text is readable without JavaScript. It cannot see firewall rules or logs.

Do I need an llms.txt file so AI bots can read my site?

No. An llms.txt file is a suggestion for AI tools, not a permission, and it does not replace robots.txt. Google says Google Search ignores llms.txt, so it neither helps nor harms visibility there ²⁰. See what llms.txt is first.

About the author

Rahil Jain

Rahil Jain is an AI SEO (answer engine and generative engine optimisation) consultant and co-founder of Chalk Labs.

LinkedIn · Method and credentials

Sources

Overview of OpenAI Crawlers · OpenAIOpened October 20, 2026. Supports: OAI-SearchBot, GPTBot and ChatGPT-User roles; opted-out sites are not shown in ChatGPT search answers; robots.txt may not apply to ChatGPT-User; about 24 hours to adjust; IP-range JSON files; robots.txt marker in the user agent; one crawl may serve both bots; version numbers may change.
Does Anthropic crawl data from the web, and how can site owners block the crawler? · AnthropicOpened October 20, 2026. Supports: ClaudeBot, Claude-User and Claude-SearchBot roles and the effect of blocking each; bots honor robots.txt and will not bypass CAPTCHAs; rules go in the top-level robots.txt once per subdomain; published IP list.
Perplexity Crawlers · PerplexityOpened October 20, 2026. Supports: PerplexityBot and Perplexity-User roles; Perplexity-User generally ignores robots.txt; published IP-range JSON files; WAF allow-listing by user agent and IP with Cloudflare and AWS WAF steps; robots.txt changes may take up to 24 hours.
RFC 9309: Robots Exclusion Protocol · Internet Engineering Task Force (IETF)Opened October 20, 2026. Supports: Group matching by user agent with the * fallback; longest-match rule and allow winning ties; 4xx means crawlers may access anything; 5xx means crawlers must assume complete disallow; robots.txt is not access authorization.
Block AI Bots and Configure AI bot policies (Bots docs) · CloudflareOpened October 20, 2026. Supports: Search, Agent and Training presets with three actions; as of the page's 1 July 2026 update, updated defaults for new domains were scheduled for 15 September 2026 (a date now passed); the older Block AI bots setting is marked as deprecating on that date.
Content Independence Day: no AI crawl without compensation · CloudflareOpened October 20, 2026. Supports: On 1 July 2025 Cloudflare announced it was changing its default to block AI crawlers unless they pay creators; the post does not say how far this reaches into existing sites.
Analyze AI traffic (AI Crawl Control) · CloudflareOpened October 20, 2026. Supports: AI Crawl Control lists crawlers with request counts and Allow or Block actions, and charts status codes including 403 blocked.
Bot Fight Mode · CloudflareOpened October 20, 2026. Supports: Bot Fight Mode cannot be skipped with WAF custom rules; Super Bot Fight Mode supports skip rules; a matching IP Access rule prevents it from triggering.
Security events · CloudflareOpened October 20, 2026. Supports: Security events are found under Analytics and the Events tab, can be filtered by user agent, and are sampled so may not show every event.
The rise of the AI crawler · Vercel (with MERJ)Opened October 20, 2026. Supports: December 2024 finding that the major AI crawlers tested did not render JavaScript while Gemini and AppleBot did; 404 shares of 34.82% (ChatGPT), 34.16% (Claude) and 8.22% (Googlebot).
Understand JavaScript SEO basics · Google Search CentralOpened October 20, 2026. Supports: Googlebot queues 200-status pages for rendering, the wait can range from seconds to longer, and server-side or pre-rendering is still recommended because not all bots run JavaScript.
Optimizing Your Content for Inclusion in AI Search Answers · Microsoft Advertising (Bing)Opened October 20, 2026. Supports: AI systems may not render hidden content such as tabs or expandable menus; avoid putting key information only in images or PDFs.
Google's common crawlers · Google Search CentralOpened October 20, 2026. Supports: Google's common crawlers obey robots.txt; Google-Extended is a standalone token with no separate user agent string and does not affect inclusion in Google Search.
AI features and your website · Google Search CentralOpened October 20, 2026. Supports: A page must be indexed and snippet-eligible to appear in AI Overviews and AI Mode with no extra technical requirements; noindex and snippet controls limit what is shown.
Disable JavaScript with Chrome DevTools · Chrome for DevelopersOpened October 20, 2026. Supports: Steps to disable JavaScript through the DevTools Command Menu.
Verifying Googlebot and other Google crawlers · Google Search CentralOpened October 20, 2026. Supports: Google crawlers can be verified by reverse DNS with forward confirmation or by matching published IP range files.
Editing robots.txt.liquid · Shopify Help CenterOpened October 20, 2026. Supports: Shopify merchants edit robots.txt through a robots.txt.liquid template; Shopify calls this an unsupported customization and suggests hiring a Shopify Partner.
Introducing AI Performance in Bing Webmaster Tools Public Preview · Microsoft Bing Webmaster BlogOpened October 20, 2026. Supports: Public preview announced 10 February 2026; reports how often a site is cited across Microsoft Copilot, Bing AI summaries and select partner integrations.
Verified bots · CloudflareOpened October 20, 2026. Supports: Cloudflare verifies bots using methods such as IP validation and Web Bot Auth.
Optimizing your website for generative AI features on Google Search · Google Search CentralOpened October 20, 2026. Supports: Google Search ignores llms.txt files, which neither help nor harm visibility there.
Generative AI performance report (Search) · Google Search Console HelpOpened October 20, 2026. Supports: The report shows impressions in AI Overviews and AI Mode; Google says it had rolled it out to all websites worldwide as of 31 August 2026.

How this page was made. Built on Rahil Jain's AI SEO method and drafted with AI research assistance. Every statistic links to the page we read it on (opened October 20, 2026 or nearby). Published October 20, 2026; last updated October 20, 2026. Chalk Labs sells AI SEO services, so we have a commercial interest in this topic. Spotted an error? Tell us and we will fix it.

How to Check If ChatGPT and Other AI Bots Can Crawl Your Website

Where can an AI bot be stopped before it reads your page?

Do it step by step

Copy-ready templates

Explicit-allow robots.txt block for AI search and user bots

Python 3: what does robots.txt say about each AI bot?

Bash: test a page with each bot's user agent

Python 3: is your text in the raw HTML?

Bash: count AI bot hits in an access log

Python 3: check bot IP addresses against published ranges

Results log to copy into a sheet

What do you need, and how long does it take?

Why check crawl access before you write more content?

Which AI bots should you look for?

How does a bot read your robots.txt file?

How can you tell if Cloudflare or a firewall is blocking AI bots?

Do AI crawlers run JavaScript?

How do you check server logs for GPTBot visits?

What mistakes waste the most time, and which gate does each break?

What did an AI assistant search for when we asked this question?

How do you know your fix worked?

When is a crawl check not worth your time?

What an AI assistant searches when it answers this

Frequently asked questions

About the author

Rahil Jain

Sources

Where can an AI bot be stopped before it reads your page?

Do it step by step

Copy-ready templates

Explicit-allow robots.txt block for AI search and user bots

Python 3: what does robots.txt say about each AI bot?

Bash: test a page with each bot's user agent

Python 3: is your text in the raw HTML?

Bash: count AI bot hits in an access log

Python 3: check bot IP addresses against published ranges

Results log to copy into a sheet

What do you need, and how long does it take?

Why check crawl access before you write more content?

Which AI bots should you look for?

How does a bot read your robots.txt file?

How can you tell if Cloudflare or a firewall is blocking AI bots?

Do AI crawlers run JavaScript?

How do you check server logs for GPTBot visits?

What mistakes waste the most time, and which gate does each break?

What did an AI assistant search for when we asked this question?

How do you know your fix worked?

When is a crawl check not worth your time?

What an AI assistant searches when it answers this

Frequently asked questions

About the author

Rahil Jain

Sources

Keep reading