Cloudflare’s AI Crawler Crackdown Catches the Bots That Behave

Cloudflare will begin blocking AI crawlers that double as search bots from every ad-supported website on its network starting September 15, 2026. The new default catches so-called mixed-use crawlers, chiefly Google’s, that use one visit to a page to feed search indexing, AI training and chatbot answers all at once.

The crawlers this rule can actually catch are the ones that already identify themselves. A far larger, unnamed scraping economy keeps running underneath it, untouched by any September deadline.

Cloudflare Takes Aim at Google’s Crawler

Starting on the September date, new Cloudflare customers and new sites from existing customers get a fresh default. So does every free-tier account that has not changed its settings. All of them will allow crawlers built purely for search, while blocking any bot classified as doing AI training or acting as an agent on pages that carry ads.

That detail quietly catches three of the biggest names in tech. Googlebot, Applebot and Microsoft’s Bingbot all mix indexing duties with AI data collection inside a single crawler, and each would lose default access to ad-supported pages once a site owner chooses to block training.

Cloudflare’s own framing makes its target explicit. The company said the largest search engine has access to roughly twice as much information as leading AI companies, since it makes it difficult for site owners to stay discoverable without also feeding its AI systems. That is the reasoning behind its promise to block AI training and agent crawlers on ad-supported pages by default.

“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” Cloudflare co-founder and chief executive Matthew Prince said in the announcement.

Cloudflare calls this its second Content Independence Day. The first, a year earlier, introduced the original one-click block on AI bots and the Pay Per Crawl marketplace, a feature the company says more than a million customers eventually turned on. September’s changes go further by setting the new rules automatically instead of waiting for site owners to flip a switch.

Bots Passed Humans While Nobody Was Looking

Cloudflare has a reason to move quickly. Its Radar service found that automated requests made up 57.4% of all HTTP traffic worldwide in early June, against 42.6% from humans, the first time bots have held a majority in the network’s records. NBC News, CNET and Tom’s Hardware all reported the same figures within days of Prince sharing them.

Prince had not expected the crossover this soon. His own forecast, given at the South by Southwest conference, had put it in late 2027.

For the millions of ordinary visitors who now hit a screen asking them to wait a few seconds before a page loads, that shift is the visible edge of the same sorting problem playing out at machine scale. Site owners are leaning harder on tools that scan a visitor’s browser for bot signals before letting anyone through.

Welp, that happened faster than I predicted.

Prince, reacting to the milestone in comments reported by Mashable, added that the web’s economics would keep shifting toward what he called pay to crawl.

Separately, HUMAN Security’s 2026 State of AI Traffic report, cited by CNBC, found that agentic AI traffic, bots completing tasks on a person’s behalf rather than just harvesting training data, grew roughly 7,851% year over year.

• 57.4% of worldwide HTTP requests are now bot-originated, versus 42.6% human, per Cloudflare Radar.
• 7,851% year-over-year growth in agentic AI traffic, per HUMAN Security’s 2026 State of AI Traffic report.
• 53% of all web traffic was automated in 2025 by a separate measure that also counts app and API calls, which found bad bots alone made up 40% of that traffic, per Imperva’s 2026 report.

The Crawl-to-Referral Math Behind the Fight

Cloudflare’s real complaint shows up in a single ratio: how many times a bot crawls a page for every human visitor it sends back afterward. A year earlier, the company measured a search crawler that crawls websites 14 times for every referral, Google’s, while OpenAI’s GPTBot sat at 1,700 to 1 and Anthropic’s ClaudeBot at 73,000 to 1.

The gap has narrowed since, though not by much. An analysis of Cloudflare Radar data by SEO analytics firm SEOmator found Google’s ratio easing to about 4.9 to 1 in the first quarter of 2026, while Anthropic’s crawler still fetched tens of thousands of pages for every visitor it referred back.

Search engines send a visitor back because their business depends on it. A generative answer does not need to, since the model already has what it needs inside its own chat window, and the user rarely clicks through to the source at all.

Does This Actually Force Google’s Hand?

Not entirely. Google already runs a separate crawler called Google-Extended that lets publishers opt out of AI training and products like Gemini without losing search visibility. Googlebot itself still powers AI Overviews and AI Mode inside core Search, so blocking it to stop AI training also means disappearing from Google’s results altogether.

Apple and Microsoft offer similar carve-outs. Applebot-Extended lets publishers keep their content out of Apple’s on-device model training while still showing up in Siri and Spotlight results, and Microsoft provides a comparable option for Bingbot.

That bundling is exactly what critics point to.

• Cloudflare and its publisher partners say separating search from training finally gives site owners real bargaining power and a realistic path to getting paid when AI systems use their work.
• David Skok, chief executive and editor-in-chief of Canadian publication The Logic, wrote in Nieman Lab that as long as Google’s search and training crawling function as one system, “the market for licensing journalism into AI models is effectively stalled.”
• Google maintains that Google-Extended already gives publishers a genuine way to block model training separately from search indexing.

For most publishers, blocking still comes down to a straight choice between staying fully visible or disappearing outright, and few can afford the second option.

The Bots Nobody Can Name

Even a perfectly enforced rule only reaches crawlers willing to say who they are. Frederick Jahn, who leads bot-defense firm Centinel Analytica, told Digiday that identified crawlers make up somewhere between a fifth and a third of bot traffic on major news sites. The rest, by his estimate, is stealth traffic dressed up to look human, routed through shared proxy networks that make it nearly impossible to tie a specific crawl back to a specific AI company.

That undeclared layer keeps growing. Bot-monetization firm TollBit found that 13.26% of AI bot requests ignored robots.txt instructions outright in the second quarter of 2025, up from just 3.3% two quarters earlier.

Blocking has kept pace anyway. Web-technology tracker BuiltWith counted roughly 5.6 million websites blocking OpenAI’s GPTBot by the end of last year, up nearly 70% from 3.3 million five months earlier.

That is also why the interstitial page many readers now see more often, the one that pauses a page for a few seconds to confirm a human is behind the click, has turned into a near-daily encounter rather than a rare one.

Publishers Are Betting Someone Will Pay

Cloudflare’s answer to the stealth economy is to make the legitimate side of the ledger worth joining. Its original Pay Per Crawl marketplace, which let publishers set a price and charge AI companies before letting a bot through, is folding into a broader system called Pay Per Use that pays publishers when their content actually shows up inside an AI answer, rather than just when a bot fetches it. Ceramic.ai and You.com are the first two partners live on the new system.

Cloudflare says its AI Crawl Control tool already issues more than a billion payment-required responses to AI crawlers every single day across its customer base, and that publishers and AI platforms have signed more than 50 major content licensing agreements over the past year.

• Condé Nast credits Cloudflare with helping ensure it gets paid as AI companies treat its journalism as training material.
• Patreon blocks known AI training crawlers across its network while still allowing the search bots that help creators get found.
• Associated Press anchors the wire-service tier of publishers already set up to charge for crawl access.
• Stack Overflow published a joint case study with Cloudflare in February 2026 arguing AI labs should pay in proportion to how much a site’s data actually contributes to a model’s ability.

Cloudflare said it would spend the two months before the deadline to “engage with the ecosystem, listen to feedback, and run tests to finalize new defaults and classifications.” The stealth crawlers already running through residential proxies were never part of that conversation.

Frequently Asked Questions

What Makes a Crawler Count as Mixed Use?

Cloudflare sorts automated traffic into three categories, Search, Agent and Training, and a crawler counts as mixed use once a single bot performs more than one of those jobs at once, exactly what Google’s, Apple’s and Microsoft’s flagship crawlers do today. Most major AI operators now run separate bots for each purpose and appear on Cloudflare’s own verified bots list, though adoption of WebBotAuth, a cryptographic standard that proves a request truly comes from the bot it claims to be, remains rare even among large AI labs.

Do Most Websites Even Have a robots.txt File?

No, not universally. Cloudflare’s own research found that only about 37% of the top 10,000 domains had a robots.txt file in place at all, the basic text file that tells crawlers which parts of a site they may or may not visit. That leaves most sites depending entirely on Cloudflare’s dashboard settings rather than any rule they published themselves.

How Do Site Owners Opt Out of the New Default?

Free-tier customers and anyone who wants different treatment than the September 15 default can adjust the block in Security Settings before the deadline, choosing separately whether to allow or block bots in the Search, Agent and Training categories.

Can an AI Company Just Ignore the Block?

Yes, in practice many already do. Newer AI-powered browsers, including Perplexity’s Comet, and developer tools like Firecrawl, run through an ordinary browser session and can look identical to a human visitor inside a site’s own server logs, according to bot-monetization firm TollBit, which makes a purely technical block impossible to enforce alone.

Are There Legal Risks to Scraping Around These Blocks?

Yes. The New York Times, Reddit and other publishers have sued AI companies including OpenAI, Microsoft, Perplexity and Anthropic over the copyright implications of training chatbots on scraped work, and courts have not yet settled whether that use counts as fair use. Deliberately circumventing a technical access control that a publisher has put up can also trigger separate liability, according to law firm Ropes & Gray.