What AI crawlers take, and what they send back

Every AI answer that names a brand began with a crawl. In the past year the operators who see that traffic at network scale, and the analytics vendors who see what happens after it, published enough data to describe the trade: how much AI systems read, how many visitors they send back, and how much of a typical site they can actually parse. This post collects those figures with their sources and dates.

Who is crawling

Cloudflare's Radar Year in Review covers traffic across its network from 1 January to 2 December 2025. Googlebot was the dominant crawler, producing over 28% of verified bot traffic and about 4.5% of all HTML requests; it accounted for 35% to 55% of crawler traffic by region in October 2025. OpenAI's GPTBot was next at about 7.5% of verified bot traffic, with volatile activity that peaked in June, and Microsoft's Bingbot at about 6%. ClaudeBot roughly doubled in the first half of 2025, declined in the second half and ended about 10% above its January level; PerplexityBot ended the year at about 3.5 times its starting level; and other AI bots together averaged 4.2% of HTML requests (Cloudflare, December 2025).

The composition changed with purpose. Cloudflare's key findings say crawling triggered by a user action, meaning an assistant fetching a page to answer a live question rather than to train a model, rose more than fifteenfold in 2025, with the detailed section putting the increase at over 21 times between January and early December (Cloudflare, December 2025). That is the category that can produce a citation and a visitor, and it is the one GEO depends on. Similarweb's reading of Cloudflare Radar data adds the 2026 trajectory: ClaudeBot's share of global AI crawler traffic rose from 9% to 20% by May 2026, with GPTBot at about 10% (Similarweb, June 2026).

Crawl-to-refer: the exchange rate

Cloudflare's crawl-to-refer ratio divides the HTML pages a platform's crawlers request by the human visits its platform sends back. In 2025 the spread between platforms was enormous. Google started just over 3:1, rose to about 30:1 in April, fell back to 3:1 by mid-July and has been slowly rising since. Microsoft's peaks sat between 50:1 and 70:1 with a weekly cycle. Perplexity had the lowest ratio of the major AI platforms, starting below 100:1, spiking above 700:1 in late March and staying below 200:1 from September. OpenAI was spiky, peaking at about 3,700:1 in March. Anthropic's ratio was the highest, peaking at about 500,000:1 and settling between roughly 25,000:1 and 100,000:1 after May (Cloudflare, December 2025).

Two factors explain why the ratios are so high and so different. First, only part of each platform's crawling is for live answers; training and index crawls read pages that never produce a visit. Second, the referral is the part of the exchange that is easiest to lose: visits from native apps often arrive without a referrer and are recorded as direct, which Similarweb flags as a reason its referral counts are a floor (Similarweb, September 2026). The ratios are therefore an upper bound on how one-sided the trade is, but the order of platforms is consistent across every measurement we found.

The citation side explains the rest. Similarweb measured that 6.8% of ChatGPT answers in the United States carried a web citation in May 2026, with travel at 22.6% and retail at 13.5%; in beauty, 54.7% of cited sources were retail or ecommerce domains (Similarweb, September 2026). Semrush found web search enabled on 34.5% of ChatGPT queries in February 2026, down from 46% in late 2024 (Semrush, April 2026). Pew's data for Google shows the pattern in search itself: 88% of AI summaries cited three or more sources, Wikipedia, YouTube and Reddit made up 15% of cited sources, and .gov sites 6% against 2% in standard results, yet users clicked a link inside the summary in only 1% of visits (Pew Research Center, July 2025).

What sites are telling the bots

Cloudflare's robots.txt analysis shows site owners separating training from search. GPTBot, ClaudeBot and CCBot received the most full-site disallow directives of any crawlers, whereas Googlebot and Bingbot directives were mostly partial, covering areas such as login pages. Explicit allow directives were far fewer than disallows overall, but allow directives for Google-Extended tripled during 2025, shifting from partial to full-site allows, and allows for OpenAI's crawlers leaned toward full-site (Cloudflare, December 2025). In 2026 Cloudflare moved the distinction into its product: Bot Preference Sync, announced in August 2026, aligns a site's robots.txt with policies for Search, Agent and Training traffic, and a September 2026 post argued that a robots.txt directive alone "cannot identify who is crawling, determine why they are crawling, or stop a crawler that ignores it" (Cloudflare, August and September 2026).

The question for a brand is no longer "block or allow AI". It is "which kind of request": a training fetch, a search-index fetch, or an agent fetching a page to answer a customer right now.

The crawler names carry the distinction. OpenAI separates GPTBot (training) from OAI-SearchBot (search index) and ChatGPT-User (live requests); Anthropic separates ClaudeBot from Claude-SearchBot and Claude-User; Google separates Googlebot from Google-Extended. A rule that blocks the training crawler but admits the search and user-action crawlers is what the Cloudflare data shows publishers converging on. Our guide to robots.txt and llms.txt covers the mechanics, and our server-log study shows how these bots behave on a real site.

Can the machine read you?

Being allowed in is not the same as being read. Adobe's AI Content Visibility Checker scores how much of a page's content a large language model can parse. In March 2026 Adobe found about a quarter of homepage and category-page content on U.S. retail sites was not optimised for LLMs and about 34% of product pages could not be properly accessed (Adobe, April 2026). By May 2026 the share of content readable by AI was 63% for cosmetics, 56% for electronics, 51% for sporting goods and apparel, 48% for grocery and 47% for furniture and home, and even in the best sectors 30% to 40% of content on high-value pages was not captured (Adobe, as reported by Digital Commerce 360, June 2026). An expanded cohort in July 2026 scored 61% LLM visibility overall, meaning 39% of homepages were not machine-readable, with apparel at 76%, electronics 70%, cosmetics 68%, sporting goods 67%, furniture and home 64%, general merchandise 63% and grocery 59% (Adobe, as reported by Digital Commerce 360, August 2026).

Adobe's own reading is that "consistent and structured product content is helping drive AI visibility" in the categories that score well, and that where an LLM cannot read pricing or specifications, "potential revenue is being left on the table" (Adobe via Digital Commerce 360, August 2026). The connection to the referral data is direct: the same Adobe dataset shows AI-referred visitors converting 60% better than other traffic in July 2026, so unreadable pages forfeit the best-converting visitors a retailer has.

The Korean crawl

Korean sites host a crawl mix no global report measures. Naver's Yeti crawler feeds the search engine that held 81.34% of Korean search on 24 May 2026 (Internet Trend via Newsway, May 2026), and AI Briefing, Naver's answer layer, reached 20% of integrated search queries with a target of about 40% (Naver via kt nasmedia, April 2026); for a Korean brand, that is the largest AI answer surface, and it is served from Naver's own index. On the global side, Korea's 2025 Internet Usage Survey found ChatGPT used by 41.8% of the population and Gemini by 9.8% (MSIT, March 2026), so GPTBot, OAI-SearchBot, ChatGPT-User and Google's crawlers matter most, with Claude and Perplexity bots behind.

Three practical consequences follow for a Korean site. First, blocking "AI" wholesale usually means blocking the user-action crawlers that serve ChatGPT and Gemini answers to the 58% of Korean search users who now use them (Opensurvey, July 2026), while leaving Naver untouched; decide per crawler and per purpose. Second, Korean ecommerce pages built as client-rendered, image-heavy templates fall into Adobe's unreadable third; the price, the specification table and the FAQ have to exist as text in the HTML the crawler receives. Third, measure the exchange yourself: count AI crawler requests in server logs against referrals received, and watch which pages are fetched after a question is asked.

What this means for Korean brands

• Analyze the crawl before the content. Server logs tell you which engines read you, which pages they fetch and whether the fetches are training or live answers. GEO Insight and Analytics start from that record rather than from a guess.
• Create pages a machine can read. Adobe's audits show the gap is often technical: text that is not in the HTML, specifications locked in images, prices rendered after load. Content Agent produces pages where the facts are in the markup, and Channel Agent distributes them to the sources engines cite.
• Measure the exchange rate. Pages crawled, answers citing you and visitors sent back are three numbers that should move together. When crawling rises and citations do not, the page is being read and not chosen; Analytics shows which of the three is the bottleneck.

Sources

1. Cloudflare, Radar 2025 Year in Review, December 2025 (crawler shares, crawl-to-refer ratios, robots.txt directives, user-action crawling growth)
2. Cloudflare, "Say it once: introducing Bot Preference Sync", 21 August 2026; "Have it both ways: stay discoverable in search while disallowing AI training", 15 September 2026
3. Similarweb, "Zero-Click Marketing", 10 June 2026 (ClaudeBot and GPTBot crawler shares, citing Cloudflare Radar)
4. Similarweb, "AI Referral Traffic by Industry: What 770 Million Monthly Visits Reveal", 3 September 2026
5. Semrush, "ChatGPT traffic analysis: Insights from 17 months of clickstream data", 7 April 2026
6. Pew Research Center, "Google users are less likely to click on links when an AI summary appears in the results", 22 July 2025
7. Adobe Digital Insights Q1 2026 data and AI Content Visibility Checker, as reported by TechCrunch, 16 April 2026
8. Adobe May 2026 readability and traffic data, as reported by Digital Commerce 360, 17 June 2026; July 2026 data, 19 August 2026
9. Internet Trend search share data, as reported by Newsway, 28 May 2026; kt nasmedia on Naver AI Briefing, 15 April 2026
10. Ministry of Science and ICT, 2025 Internet Usage Survey, 31 March 2026 (as reported by Newspim); Opensurvey, AI Search Trend Report 2026 H2, 20 July 2026
11. flowgen, "What we discovered about AI crawlers via server log analysis"; flowgen, "robots.txt and llms.txt: what they do and what they do not"