Automated traffic now accounts for 57.5% of HTML web traffic, surpassing human activity for the first time, according to data shared by Cloudflare CEO Matthew Prince on June 3, 2026.
The Cloudflare Radar data showed that bots generated 57.5% of HTML requests, compared with 42.5% from human users, marking a significant shift in how the internet is accessed and consumed.
Prince acknowledged the unexpected pace of the change, saying, “Welp, that happened faster than I predicted.”
AI crawlers drive growing bot traffic
The findings point to the growing influence of artificial intelligence (AI) crawlers, which collect online content to train AI models and power AI-driven services. However, the data suggests that much of this activity does not translate into meaningful referral traffic for publishers whose content is being accessed.
According to Cloudflare, AI crawlers accounted for 20.3% of verified bot traffic in May 2026, while AI-search bots contributed another 6.5%. Together, AI-related activity represented approximately 26.7% of verified bot traffic.
Most AI crawling serves model training
The data also showed that most AI crawling was directed towards model training rather than search. Training accounted for 51.8% of AI crawler requests in May, while 35.7% served mixed training and retrieval purposes. Search-only activity represented just 9.3%, although it had increased from 7.5% in April.
This distinction has significant implications for publishers, as training crawlers typically consume online content without directing users back to the original websites.
Publishers face an uneven traffic exchange
The imbalance is reflected in crawl-to-referral ratios, which measure how many pages a crawler accesses for every referral visit it generates. According to SEOmator’s analysis of Cloudflare Radar data, ClaudeBot crawled 23,951 pages for every referral in the first quarter of 2026. The ratio improved to 11,122 pages during the week of May 25 to June 1.
By comparison, OpenAI’s GPTBot recorded a ratio of 1,276:1, while PerplexityBot stood at 111:1. Traditional search engines generated considerably more referral traffic, with Google recording a ratio of 4.9:1 and DuckDuckGo 1.5:1.
The figures highlight the growing disconnect between the volume of content accessed by AI systems and the traffic returned to the websites producing that content.
Websites step up efforts to block AI bots
The trend has prompted publishers and website operators to introduce measures restricting AI access. Cloudflare made AI-crawler blocking the default setting for new domains from July 1, 2025. In the following five months, its customers blocked 416 billion AI bot scraping requests, according to figures shared by Prince.
More than 2.5 million websites had opted to fully disallow AI training as of August 2025.
The issue has also raised questions about the ability of publishers to restrict AI access without affecting their visibility on conventional search platforms.
Referring to Google’s combined search and AI crawling systems, Prince said, “You can’t opt out of one without opting out of both, which is crazy.”
AI referrals bring higher-value visitors
Meanwhile, AI-generated referral traffic, although relatively small, is emerging as a potentially valuable source of visitors for some businesses.
Adobe reported that AI-referred traffic to US retailers increased 393% year-on-year in the first quarter of 2026. Visitors arriving through AI platforms converted 42% better than those from non-AI channels, generated 37% more revenue per visit and spent 48% more time on websites as of March 2026.
However, the findings are specific to the retail sector and should not be interpreted as representative of all websites.
The broader shift presents publishers with two challenges: deciding which AI crawlers should be allowed to access their content and developing better ways to measure traffic and engagement generated through AI platforms.
The data suggests that as AI systems increasingly consume online content for training and retrieval, traditional measures of website visibility and traffic may no longer adequately capture the value publishers create or the returns they receive.
