Cloudflare’s September 15 Deadline Forces AI Crawlers Apart

Cloudflare will block AI training and agent crawlers by default on ad-supported pages starting September 15, 2026, the internet infrastructure company said on Wednesday, ending a year of voluntary opt-outs and replacing them with a presumption that publishers want nothing from bots they cannot bill. The shift, framed by Cloudflare co-founder and CEO Matthew Prince as the second “Content Independence Day,” applies to all new Cloudflare customers, every new site created by existing customers, and every free-tier account from that date onward. The full announcement lives in Cloudflare’s rules update on AI traffic options.

The change is built around a new taxonomy that splits AI bots into three categories, search, agent, and training, and forces the largest crawlers, including Googlebot, Applebot, and BingBot, to declare which work they do on a publisher’s pages. Cloudflare’s own data, laid out in Prince’s Content Independence Day essay, argues that bots now drive the majority of web traffic for the first time, a milestone the company did not expect to arrive until 2027.

What the September 15 Deadline Actually Does

On September 15, the default configuration for any new domain joining Cloudflare, any new site an existing customer sets up, and any site still on Cloudflare’s free tier will block two of the three new bot categories on pages that display advertising. Training crawlers, the bots that copy content to feed model training, and Agent crawlers, the bots that act on behalf of a human user in real time, will be denied by default. Search crawlers, the bots that build indexes for traditional search results, will continue to be allowed.

The defaults can be overridden by any site owner, and Cloudflare said it would notify customers ahead of the change. But the direction of travel is the point. A year ago, Cloudflare shipped a one-click “Block AI bots” toggle, and more than one million customers turned it on. The September 15 update removes the need to click.

Three bot categories will be governed by Cloudflare’s new defaults on September 15, 2026:

• Search crawlers: allowed by default on ad pages
• Training crawlers: blocked by default on ad pages
• Agent crawlers: blocked by default on ad pages

For multi-purpose crawlers, those that bundle more than one behavior into a single bot, the strictest applicable rule wins. Cloudflare named the three largest of them in the announcement: Googlebot, Applebot, and BingBot. A customer who selects “block Training” will therefore also lose access to the search side of those bots, unless the publisher opts out of the new defaults entirely.

The Three Categories Cloudflare Now Draws

The taxonomy is the heart of the policy. Cloudflare is replacing a single binary, is this bot an AI bot, yes or no, with a behavioral classification built on three questions: what is the bot doing on the site, what does it keep, and how does it reshare what it finds.

Cloudflare is also introducing a secondary axis, a “content use” signal, that lets publishers set one of three levels: immediate (interact but store nothing), reference (index, excerpt, link back, the new default), or full (summarize and reproduce). The signal will be added to robots.txt as a new field called “use,” extending Content Signals, the standard Cloudflare helped launch last year, with details logged in the Cloudflare developer changelog for AI traffic options. Site owners who already use Cloudflare’s managed robots.txt will see “use=reference” added automatically.

Why Google Is Named, and Why That’s a Trap

Cloudflare named Google in the announcement, and not in passing. The essay accuses the world’s largest search engine of holding roughly twice the informational reach of every other AI company, because publishers cannot stay discoverable on Google without also consenting to Google’s AI use. Cloudflare’s framing: customers face a “Faustian bargain” in which the only way to remain in Google Search is to let Google train and serve AI on the same content.

Google has pushed back on that framing before, and did so again this week. The company’s position, set out in its Google Extended opt-out tool, is that publishers can already block Google from training Gemini and feeding its other AI products without losing Google Search. Googlebot crawls for Search, including AI features like AI Overviews and AI Mode. Google Extended governs only the AI training and product side. The two, Google argues, are separable.

Cloudflare’s new defaults make that separation harder, not easier. Because Googlebot is classified as a multi-purpose crawler under the new taxonomy, any publisher who flips “block Training” to on will, by default, lose Googlebot too. To keep showing up in Google Search while also blocking Google from training on the same content, a publisher will now have to actively opt out of Cloudflare’s new defaults. The new model shifts the burden from Google, which once had to argue for permission, to every publisher, which now has to argue for carve-outs.

The dispute over Google’s bundled bot sits inside a broader shift in how AI surfaces ads, including the five ad formats Google now runs inside Gemini answers. The Google dispute is the part of the policy that could draw regulatory attention in Europe, where competition authorities have already opened files on Google’s AI Overviews. Cloudflare’s argument that Google holds “2x more information” than other AI companies is, in effect, an accusation of self-preferencing in the crawlers themselves. Whether that accusation survives contact with the European Commission’s existing search investigation is a question for September, not July.

The Traffic Math Driving the Fight

The numbers behind Cloudflare’s escalation are blunt, and the company laid them out in Prince’s essay. The case for Content Independence Day is built on the gap between how much content AI companies take and how much traffic they send back, compared with the Google of twenty years ago.

• 75%: share of mobile Google queries now answered without the user leaving Google, per research cited by Cloudflare.
• 750x: harder for publishers to get traffic from OpenAI than from Google at its 2004 peak, per Cloudflare’s own analysis.
• 30,000x: harder for publishers to get traffic from Anthropic than from Google at its 2004 peak, per Cloudflare’s own analysis.
• 2x: more information available to Google than to any other AI company, because search discoverability and AI consent are bundled, per Cloudflare.
• 50%+: share of AI crawler traffic that consists of re-fetches of pages that have not changed, per Cloudflare.

The 50% figure is the one Cloudflare’s engineering team is leaning on hardest. It implies that half of what AI crawlers spend their bandwidth on is wasted work, repeat visits to the same page. From a publisher’s perspective, that wasted bandwidth also bills for compute. From Cloudflare’s perspective, it justifies the cost of a default block: most of what is being fetched is not new, and most of what is new is not being paid for.

The 750x and 30,000x figures are harder to verify outside Cloudflare, and the company has not published the underlying methodology. Anthropic and OpenAI have not, in this round, disputed the order of magnitude. Both companies have said they support publisher compensation in principle. Neither has said how, or how much.

The broader shift that put Cloudflare in this position, automated traffic crossing the majority of all web requests, is detailed in how anti-bot protection became core web infrastructure. Cloudflare treats that milestone as the trigger for the September 15 escalation; AI companies are likely to treat it as the ceiling of what they can scrape without paying.

Pay Per Crawl Becomes Pay Per Use

Cloudflare is not just turning crawlers away. It is also rebranding the marketplace it launched a year ago. Pay Per Crawl, which billed an AI company every time its bot fetched a page from a willing publisher, is becoming Pay Per Use, which bills an AI company every time its content creates value downstream, including when a publisher’s article appears in an AI search result.

Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.

The shift matters because it puts the price on the answer, not the crawl. A publisher whose article shows up as a citation in a Ceramic.ai search summary can now get paid even if Ceramic’s crawler never re-fetched the page in question. Cloudflare is initially working with two partners on the new model: Ceramic.ai, an AI search engine, and You.com, a consumer search product that already licenses premium content. Other AI companies will be able to customize the model for their own products.

Prince wrote that a sustainable ecosystem cannot emerge under the old voluntary regime, and the September 15 deadline is the line Cloudflare drew between experiment and policy. Whether publishers, in volume, opt out of the defaults and keep their existing settings, or let the new blocks stand, is the test that begins on that date.

What Slips Past the New Defaults

The new policy has carve-outs, and they are visible. Site owners can opt out of the defaults entirely and keep their existing bot settings, and Cloudflare said it would notify every customer ahead of the change. Paid Cloudflare customers on existing plans will not see the new defaults applied retroactively to sites they already operate; only new sites they create will fall under the September 15 rules. That alone protects a large share of the publisher base, which is on paid plans and was set up before this week.

The defaults also do not apply to non-ad pages. A publisher who runs an ad-free archive, a developer documentation site, or a paywalled product will see Training and Agent bots continue to be allowed by default, on the theory that those pages are not trying to monetize human attention. Cloudflare’s framing is that an ad is the publisher’s signal that a human visit was the intended outcome, so the policy only switches on where that signal is present.

Search engines remain allowed on every page, including ad pages, because they are still treated as the publisher’s main funnel for human visits. That carve-out is what makes the policy survivable for most news and media sites, and it is also the reason the Google dispute is so loud: Googlebot is the only multi-purpose crawler in the world that, if blocked, removes a publisher from the dominant search index.

Two new Cloudflare products ship alongside the policy. BotBase is a searchable directory of every bot Cloudflare tracks, with each bot classified against the new taxonomy and a detection ID that customers can paste into security rules. Enterprise Bot Management customers get it first, with a wider rollout promised later this year. Cloudflare is also opening a waitlist for what it calls the Monetization Gateway, a layer that lets any publisher charge for any web page, dataset, API, or MCP tool behind Cloudflare, with settlement in stablecoins over the x402 open protocol.

The structural bet behind all of it is that the next decade of the web will be settled through Cloudflare, not negotiated between each publisher and each AI company. Prince made that bet publicly when he framed his announcement as a declaration rather than a product update. Bot traffic crossed 50% of total web requests earlier than the company expected, and the September 15 deadline is the first major policy built around the assumption that the milestone has already passed.