Skip to content
Chalk Labs / AI SEO
How-to guide

How to Speed Up Your Site for AI Crawlers

By Rahil Jain, founder of Chalk LabsUpdated How we research

The short answer

No public study we can find shows that faster pages get cited more by AI assistants. What is documented is narrower: Google crawls less when your server slows down or returns errors, and it uses Core Web Vitals in ranking. So fix errors and slow server responses first, because they can cost you crawling, and do not expect speed alone to win citations.

  • We found no public study linking page speed to AI citations. Do not trust anyone who gives a number.
  • Google documents that it crawls less when response times lengthen or the server returns errors.
  • Fix errors first: 5xx server errors, 429 rate limits and broken links, then time to first byte (TTFB).
  • Crawl budget only bites on large sites. Most small sites should treat speed as a user and reliability job.

Does page speed affect AI crawling and citations?

For citations, no evidence either way exists in the sources we could open. Google says Core Web Vitals are used by its ranking systems, but that good scores do not guarantee top rankings and that it still shows the most relevant content when page experience is poor 1. That page covers Google Search ranking. It says nothing about ChatGPT, Perplexity or Copilot citations.

For crawling, the evidence is firmer but limited to Google. Its crawl budget guide says the crawl capacity limit goes up when response times, including latency and time to first byte, stay stable or improve, and goes down when the site slows or returns server errors or rate-limiting signals such as HTTP 429 2. The other AI companies do not publish an equivalent rule that we could verify.

So the honest summary is this. Speed protects your crawling on Google. For other assistants it is a sensible precaution, not a proven lever.

What is known and what is not
ClaimStatusSource
Googlebot crawls less when response times lengthenDocumented by GoogleGoogle crawl budget guide 2
Google uses Core Web Vitals in rankingDocumented, with the limits aboveGoogle page experience guide 1
Faster pages are cited more often by ChatGPT or PerplexityNo public study foundNone
AI bots slow down when your server is slowNot documented by the AI companies we checkedNone
Server errors can stop crawling for hoursDocumented for Google's robots.txt handlingGoogle robots.txt guide 3

Do it step by step

0 of 7 done0%

  1. Choose your home page, two service or product pages, two blog posts, your pricing page, your robots.txt and your sitemap. These are what bots hit first.

  2. Run the timing command in the templates below for each URL, three times, at different times of day. Write down the status code and the TTFB each time.

    Tip: Use your worst reading, not your best, as the number you try to improve.

  3. Run the log command below on the last 30 days of access logs. Any 429 or 5xx next to a bot name needs an owner and a fix.

    Tip: If you cannot get server logs, ask your host or your CDN for a bot report.

  4. Resolve server errors and timeouts first. Then remove rate limits that hit search bots, then redirect broken URLs. Retest the eight URLs after each change.

  5. Cache HTML for pages that do not change per visitor. Serve it from a CDN. Use short cache times such as five minutes for news pages and longer for evergreen pages.

  6. Link directly to final URLs. Make sure each redirect is one hop. Check that raw HTML is well under 2MB and that large scripts are loaded separately.

  7. Repeat steps two and three. Save the before and after table. If TTFB is under 0.8 seconds and bot errors are near zero, stop and move on to content.

Your ticks are saved in this browser only.

Copy-ready templates

Time to first byte and status check (bash)

bash
for url in \
  https://example.com/ \
  https://example.com/services \
  https://example.com/robots.txt \
  https://example.com/sitemap.xml
do
  curl -o /dev/null -s -L \
    -w "$url status=%{http_code} ttfb=%{time_starttransfer}s total=%{time_total}s\n" "$url"
done

Replace example.com. ttfb is the time until the first byte arrives. Run it three times and keep the worst figure.

Status codes returned to AI bots (access log, combined format)

bash
grep -E "OAI-SearchBot|GPTBot|ChatGPT-User|PerplexityBot|ClaudeBot|Claude-SearchBot" access.log \
  | awk '{print $9}' | sort | uniq -c | sort -rn

Assumes the common combined log format, where the ninth field is the status code. Adjust the field number if your format differs.

Cache header for pages that rarely change

text
Cache-Control: public, max-age=300, stale-while-revalidate=600

An example only: the page may be served from cache for 5 minutes and refreshed in the background for 10 more. Set values to suit how often you edit.

Speed and access checklist

markdown
- [ ] robots.txt returns 200 and loads quickly
- [ ] No 5xx or 429 responses to search bots in the last 30 days
- [ ] Key pages have TTFB under 0.8s at the worst test time
- [ ] Old URLs redirect in one hop
- [ ] HTML is cached or static where possible
- [ ] Raw HTML of key pages is well under 2MB
- [ ] Firewall rules reviewed so search bots are not blocked by accident

Review monthly.

Does your site have a crawl budget problem?

Probably not, unless it is large. Crawl budget is the amount of crawling a search engine is willing to do on your site. Google says its guide is mainly for sites with 1 million or more unique pages that change weekly, or 10,000 or more that change daily, and calls these thresholds a rough estimate 2.

If you run a 60-page service site, speed still matters for visitors, but crawl budget is not the reason. Spend your effort on content and access instead. If you do sit near those thresholds, the guide to monitoring AI bot activity in server logs shows what bots actually fetch.

AI crawlers can also generate heavy load. Cloudflare's 2025 review found that crawling for model training made up the vast majority of AI crawler traffic, peaking at 7 to 8 times search crawling 4. That is a reason to keep servers healthy, and a reason some owners rate-limit bots, which the next section covers.

What should you fix before speed?

Fix errors before speed. A fast error page helps nobody. Three error types do the most damage: server errors, rate limits that hit real bots, and links that return 404.

Server errors matter most. Google says that if your robots.txt returns a 5xx error, it stops crawling the site for the first 12 hours while it keeps retrying, then uses the last good copy for up to 30 days 3. A flaky server at the wrong moment can pause crawling of the whole site.

Rate limits come next. Google lists HTTP 429 as a signal that lowers its crawl limit 2. If you set a firewall to slow bots, test that it does not block the search bots you want. Allow OAI-SearchBot and similar bots deliberately; see how to allow AI crawlers in robots.txt.

Broken links are third. Vercel found 34.82% of ChatGPT crawler fetches and 34.16% of Claude crawler fetches hit 404 pages, against 8.22% for Googlebot, in data from late 2024 5. Keep old URLs redirecting to their new homes.

What speed numbers should you measure?

Measure time to first byte (TTFB) and status codes. TTFB is the time from the request to the first byte of the reply arriving. web.dev gives 0.8 seconds or less as good and over 1.8 seconds as poor, and calls that a rough guide, not a Core Web Vitals metric 6.

Test from outside your office network, at different times of day, on several page types. Look at the worst runs, not the average. A site that is fast at 10am and slow during a backup at 3am will lose crawls at 3am.

In server logs, count the status codes returned to AI bots. A healthy site shows mostly 200 and 301. A site in trouble shows 429, 500, 502, 503 or 504 in any quantity.

How do you reduce TTFB?

Start with hosting, then add a content delivery network (CDN), a network of servers that stores copies of your pages near visitors. web.dev says hosting should come first, and a CDN should be a high priority if you do not use one 7. It also recommends caching at the edge, removing needless redirects, and pre-building static HTML for pages with no per-user content 7.

Static pages are the biggest single win for most content sites. The server sends a ready file, so it does no work per request. web.dev notes static rendering gives a consistently fast TTFB, while server rendering can raise it because pages are built per request 8. The guide on server-side rendering covers when to choose each.

Do not chase a perfect score. Google says a good page experience score does not guarantee top rankings 1. Stop when TTFB is steady under the 0.8-second guide on your key pages.

What mistakes do site owners make?

The first mistake is blaming speed for a visibility gap that has another cause. If an assistant never names you, check access, content and sources before spending on servers. Our brand audit guide shows how to find the real gap.

The second mistake is blocking bots in a rush to cut load. A rule meant for training crawlers can also catch search bots. Keep search bots and training bots as separate decisions.

The third is oversized pages. Google reads only the first 2MB of a supported file type, measured uncompressed 9. Trim inline scripts and huge embedded data so your text is not pushed past the cut-off.

When is speed work not worth it?

Skip dedicated speed work if your pages already respond in under a second, your logs show clean 200 responses to bots, and your site is small. Your time earns more in better answers and clearer pages.

Do the access check instead: check whether AI bots can crawl your site or run the free AI crawler checker.

Frequently asked questions

Does page speed affect AI citations?

We found no public study showing it does. Google says Core Web Vitals are used in ranking, but that good scores do not guarantee top rankings 1. Treat speed as protection for crawling and visitors, not a proven citation lever.

Do AI crawlers have a crawl budget?

Google documents a crawl capacity limit for Googlebot that falls when response times lengthen or errors appear 2. The AI companies we checked do not publish an equivalent rule, so assume nothing and watch your own logs.

What is a good TTFB?

web.dev treats 0.8 seconds or less as good and above 1.8 seconds as poor, as a rough guide 6. TTFB is not a Core Web Vitals metric, so use it as a health check on the server.

Should I block AI bots to reduce server load?

Only if load is a real problem, and then block by purpose. Cloudflare found training crawls were the vast majority of AI crawler traffic in 2025 4. Blocking training bots is a different decision from blocking search bots, which can remove you from AI answers.

Is a CDN worth it for a small site?

Often yes. web.dev calls a CDN a high priority for TTFB if you do not use one 7. Many hosts include one. Check your host before paying for a separate service.

About the author

Rahil Jain

Rahil Jain is an AI SEO (answer engine and generative engine optimisation) consultant and co-founder of Chalk Labs.

LinkedIn · Method and credentials

Sources

  1. Understanding page experience in Google Search results · Google Search CentralOpened October 20, 2026. Supports: Core Web Vitals are used by Google's ranking systems; good scores do not guarantee top rankings; Google shows the most relevant content even if page experience is sub-par.
  2. Managing crawl budget for large sites · Google Search CentralOpened October 20, 2026. Supports: Crawl capacity rises with stable or improving response times and falls with slower responses, 5xx errors or HTTP 429; the guide targets sites with 1M+ pages or 10,000+ pages with daily changes, as a rough estimate.
  3. How Google interprets the robots.txt specification · Google Search CentralOpened October 20, 2026. Supports: On 5xx errors for robots.txt Google stops crawling the site for the first 12 hours, then uses the last good version for up to 30 days.
  4. Cloudflare Radar 2025 Year in Review · CloudflareOpened October 20, 2026. Supports: Training crawls were the overwhelming majority of AI crawler traffic in 2025, up to 7-8x search crawling at peak.
  5. The rise of the AI crawler · Vercel (with MERJ)Opened October 20, 2026. Supports: Published 17 Dec 2024: 34.82% of ChatGPT crawler fetches and 34.16% of Claude crawler fetches landed on 404 pages vs 8.22% for Googlebot.
  6. Time to First Byte (TTFB) · web.dev (Google)Opened October 20, 2026. Supports: TTFB definition; good is 0.8 seconds or less, poor is over 1.8 seconds as a rough guide; not a Core Web Vitals metric.
  7. Optimize Time to First Byte · web.dev (Google)Opened October 20, 2026. Supports: Hosting first, CDN as a high priority, edge caching, removing redirects, pre-built static HTML for pages without per-user content.
  8. Rendering on the Web · web.dev (Google)Opened October 20, 2026. Supports: Static rendering achieves a consistently fast TTFB; server-side rendering can increase TTFB because pages are generated per request.
  9. Googlebot · Google Search CentralOpened October 20, 2026. Supports: Googlebot crawls the first 2MB of a supported file type, measured on uncompressed data.
How this page was made. Built on Rahil Jain's AI SEO method and drafted with AI research assistance. Every statistic links to the page we read it on (opened October 20, 2026 or nearby). Published October 20, 2026; last updated October 20, 2026. Chalk Labs sells AI SEO services, so we have a commercial interest in this topic. Spotted an error? Tell us and we will fix it.