Cloudflare’s September 15 Deadline Puts AI Crawlers on the Clock

Cloudflare will start blocking artificial intelligence crawlers that blend search and training duties from every ad-supported page on its network starting September 15, 2026. The deadline lands on a company that quietly runs traffic and security checks for roughly a fifth of the world’s websites. Its own numbers now show automated visitors outnumbering people across large parts of the internet it can see.

That trade-off plays out one page load at a time, inside the same verification screen that flashes before a click resolves into a webpage. On many sites, the script quietly checking a visitor’s browser is running the exact sorting logic this fight is actually about. Come September, the rules behind that screen get a lot more specific.

Cloudflare Sets a September 15 Deadline for AI Crawlers

The policy sorts automated visitors into three buckets. Search bots index a page to answer questions about it later. Agent bots act in real time on a person’s behalf, the software behind a chat assistant browsing on someone’s request. Training bots absorb content permanently into a model’s weights.

New domains joining Cloudflare, new sites from existing customers, and every account still on the free tier will default to allowing Search crawlers while blocking Training and Agent crawlers on any page that carries advertising, part of the broader reset of Cloudflare’s bot defaults taking effect that day. Cloudflare confirmed the new defaults in a blog post marking one year since it first let customers block AI bots with a single click, a move it called Content Independence Day.

A handful of concrete changes take hold at once.

• New and free accounts – any site signing up after September 15, plus every free-tier customer who has not changed settings, switches automatically to the new defaults.
• Bundled crawlers – a bot that mixes search with training, including Googlebot, Applebot and Bingbot, gets blocked wherever a site owner has chosen to block Training, even on pages where Search stays allowed.
• Opt-outs – existing customers can keep their current setup by changing their Security settings any time before the deadline.
• Pay Per Use – Cloudflare’s crawler marketplace, previously called Pay Per Crawl, starts paying publishers when their content actually surfaces inside an AI answer rather than simply when a bot fetches the page.

Cloudflare says the change should also cut waste. More than half of all AI crawler requests, by its own measurement, go toward re-fetching pages that have not changed since the last visit.

Why Machines Started Outnumbering Us Online

The urgency traces to one number. On June 3, Prince pointed to Cloudflare’s internal tracking: automated systems generated 57.5% of HTTP requests, the basic exchanges that load a webpage, to web content that month, against 42.5% from people. It was the first time the company had recorded machines in the majority.

He had not expected that so soon. Months earlier, at a conference in Austin, Prince had put the crossover about a year further out. “The majority of traffic on the Internet is non-human,” he said, arguing the industry has to move faster to keep things workable for everyone.

Researchers point to agentic AI as the main driver, the autonomous programs that browse on behalf of assistants like ChatGPT or Gemini. A single agent can visit thousands of pages to finish a task a person would complete in a handful of clicks.

A separate benchmark points the same direction. Cybersecurity firm HUMAN Security found automated traffic growing eight times faster than human traffic through 2025, with AI agent traffic specifically up nearly 7,851% year over year.

Cloudflare’s live bot-versus-human traffic dashboard is the source most coverage points back to, though not everyone reading it lands on the same conclusion.

Three separate readings of similar data are circulating right now, and they disagree on how far along this shift actually is.

• Cloudflare’s headline figure – Prince’s 57.5% number covers HTTP requests specifically for HTML content, and it is the figure driving the September 15 policy.
• A narrower dashboard reading – a separate pull of Cloudflare Radar data for the week of June 15 to 22, 2026, covering all HTTP requests rather than just HTML pages, put bot share at 35.2%, with humans still at 64.8%.
• An outside security sample – Imperva’s 2026 report found automated traffic at 53% of all web traffic in 2025, using a different, attack-surface-weighted method entirely.

All three can be true simultaneously, since each measures a different basket of traffic.

Is Google Really Cloudflare’s Target?

Yes, though Cloudflare never names the company outright. Googlebot performs search indexing and AI-training collection inside one crawler, and Cloudflare’s own announcement says the “world’s largest search engine” holds roughly twice the information access of other AI companies, a clear enough description of who that is.

Apple and Microsoft run similar dual-purpose bots through Applebot and Bingbot. Google is the one Cloudflare keeps circling back to. The company’s chief executive has previously pushed regulators to force Google into splitting its crawler into separate parts for search and for AI.

Google already offers a partial fix. Its Google-Extended tool lets a site opt out of AI training without losing search inclusion. That opt-out does not cover Google’s own AI Overviews or AI Mode results, which still draw on the same index Googlebot builds.

So a publisher blocking Training crawlers on Cloudflare after September 15 blocks Googlebot too, on any page carrying ads, unless that publisher manually opts out of the new default.

The Crawl-for-Referral Deal Publishers Say Is Broken

Cloudflare describes the old arrangement in blunt terms. Crawlers indexed a site, and the site got readers back in return. That exchange held for about three decades, Cloudflare says, and it no longer does, because AI crawlers increasingly take content without sending traffic back.

The numbers back that up unevenly. Google crawls about 14 pages for every referral visit it sends back, among the tightest ratios of any major bot. Training-only crawlers look nothing like that.

Website owners have been trying to police this on their own for years, mostly without much luck. Analytics firm BuiltWith found the number of sites disallowing OpenAI’s GPTBot in their robots.txt files climbed to about 5.6 million by December 2025, up nearly 70% from 3.3 million five months earlier. Applebot faced a similar climb, and roughly half of all news sites were blocking GPTBot outright by that point.

A report from bot-traffic firm TollBit found close to 30% of AI bot scrapes ignored a site’s disallow instructions anyway. Robots.txt was always voluntary. Nothing in it stops a crawler that decides not to read it.

That is part of why Cloudflare revived a status code that had sat almost unused for decades. Its AI Crawl Control system sends a customizable 402 response, Payment Required, letting a publisher name a price before a crawler gets the page.

Small Publishers Are Still Locked Out

Cloudflare’s tools assume a publisher has the staff and time to configure them. Many do not.

Some can even pursue direct licensing deals with AI companies.

Most others cannot, said Dries Buytaert, the founder of the open-source platform Drupal, in Cloudflare’s original announcement of the policy a year ago. Large publishers can afford to build detection systems or negotiate their own terms. Everyone else is left hoping a marketplace built by someone else works out in their favor.

The strain shows up hardest in places with no ad budget to defend at all. The Wikimedia Foundation, which runs Wikipedia, reported that bandwidth for multimedia surged 50% since January 2024, and roughly 65% of its most resource-intensive traffic came from bots even though bots made up only about 35% of pageviews.

When hosting company Kinsta analyzed more than 10 billion requests across its own managed infrastructure, the trouble was rarely sheer volume. It was repetition, crawlers hammering the same dynamic pages, like shopping carts, over and over for no obvious reason.

Referral traffic tells a similarly lopsided story. Raptive, which represents a network of 6,000 independent publishers, has said ChatGPT drives less than 0.2% of its members’ traffic, even as those publishers absorb the server cost of being crawled anyway.

What Happens After September 15

Google has not said whether it will split Googlebot before the deadline. Neither Apple nor Microsoft has said whether Applebot or Bingbot will change either. All three currently offer opt-out tools that stop short of a full separation.

Cloudflare says mixed-use crawling, which it puts at roughly a third of all crawler activity today, should fall to zero by the middle of 2027. Reaching that would require every major AI company to run separate, clearly labeled bots for search, agents and training.

Whether any of this turns into real money for newsrooms depends heavily on Google specifically. One industry analysis published late last year argued that as long as Google’s search and training crawlers stay bundled, the broader market for licensing journalism into AI systems remains effectively stalled.

Publishers are also pushing further up the supply chain. In early June, industry group Digital Content Next sent the nonprofit Common Crawl a cease-and-desist letter on behalf of the Associated Press, The New York Times and other members, demanding it stop scraping their content and remove what it had already collected.

The Interactive Advertising Bureau’s standards arm, IAB Tech Lab, ran its own bot-management guidance through a public comment period that closed June 25, telling publishers that blanket blocking is no longer a viable strategy by itself.

Uptake on the payment side remains the biggest open question. Only two companies, AI search platforms Ceramic.ai and You.com, are named as launch partners for Pay Per Use, and Cloudflare has not said how much revenue Pay Per Crawl generated during its first year in beta.

Whatever changes, the checkpoint stays. The browser check keeps flashing on countless pages long after September 15 passes. The only real difference is what it has quietly been told to let through.