Cloudflare’s Anti-Crawler Reset Hits the AI Industry on September 15

The page that greeted a routine web request on Tuesday, July 7, 2026, looked ordinary enough. “Anti-Crawler Protection is checking your browser and IP,” the message read, before announcing a three-second countdown and an automatic redirect. That interstitial screen, the kind Cloudflare and its peers serve billions of times a day, is the visible edge of a structural shift now reaching its sharpest point. By September 15, 2026, Cloudflare’s default settings will block mixed-use AI crawlers from any website that hosts ads, forcing the artificial intelligence industry to pay for what it has been taking for free.

What the three-second check protects is no longer just a website. It is the cost of producing the content the crawlers are harvesting. The reckoning that arrived in policy memos this week was set in motion years earlier, when the first large language models vacuumed up the open web and publishers realised they had no way to charge for it.

Cloudflare Sets a September 15 Deadline

Cloudflare announced on July 1, 2026 that starting September 15, its default settings would block crawlers that blend search, agent use, and training from any site running advertising, unless a site owner adjusts the configuration. The change applies to new Cloudflare customers, new sites set up by existing customers, and all existing free customers. Free-tier site owners, who make up the bulk of Cloudflare’s network, will see the new defaults without lifting a finger.

“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” Cloudflare co-founder and CEO Matthew Prince said in the announcement. The statement refers to a milestone that arrived ahead of schedule: bots now exceed humans on the open web, and the milestone was not expected until 2027.

Cloudflare is also turning its existing Pay Per Crawl marketplace into something it calls Pay Per Use. The original Pay Per Crawl programme, launched in 2025, let publishers charge AI bots per page fetched. Pay Per Use goes further, charging AI companies when content produces value downstream, not just when the URL is requested. Cloudflare is initially working with two partners on the model: Ceramic.ai, an AI search engine, and You.com, which will pay publishers when it accesses premium content.

The Web Is Now More Bot Than Human

The scale behind the policy shift comes from the Imperva Bad Bot Report, which has tracked automated traffic since 2013. In 2024, automated traffic reached 51 percent of all web requests, the first time bots surpassed humans on the open web. Malicious bots alone hit 37 percent of all traffic, up from 32 percent the year before. In 2026, the share of automated requests climbed further, to more than 53 percent of all web activity, according to figures cited by Ficstar and PromptCloud.

Those numbers describe the shape of the problem Cloudflare is selling against. Anti-bot infrastructure has shifted from a security add-on to core platform work, alongside hosting, storage, and content delivery. The bot mitigation market is projected to grow from $0.9 billion in 2025 to $1.12 billion in 2026, reaching roughly $2.4 billion by 2030, according to The Business Research Company. Gartner research cited in the same space estimates poor data quality, much of it bot-driven, costs organisations an average of $12.9 million per year.

Inside the AI Labyrinth Honeypot

The defensive toolkit has moved well beyond simple IP blocks. In March 2025, Cloudflare introduced AI Labyrinth, an opt-in feature that serves AI-generated decoy pages to crawlers it identifies as unauthorised. The pages are realistic enough to look like a real website but contain no real content from the protected site. Crawlers that follow the maze waste their own compute budgets, and any visitor that navigates four links deep is almost certainly not human.

Cloudflare built the labyrinth on its own developer platform, generating the decoy content in advance using Workers AI and an open-source model, then storing it in R2 object storage for fast retrieval. The pre-generated pages carry meta directives that block search engine indexing, so the traps never leak into Google or Bing. The feature is available on an opt-in basis to all customers, including the Free plan.

The technique is a modern take on honeypots, an idea dating back to the late-1986 Cuckoo’s Egg incident. Cloudflare’s founders ran Project Honeypot in 2004 before founding the company. AI Labyrinth is the same idea at web scale: serve the bot something it cannot tell is fake, then use the signal to improve the broader detection model. Every crawler that falls in feeds fresh training data into Cloudflare’s bot classification systems.

The New Economics of Crawling

Cloudflare’s data points to a wasteful pattern on the AI side of the trade. According to the company’s measurements, more than 50 percent of crawl traffic from AI crawlers is spent re-fetching pages that have not changed since the last visit. The wasted requests burn publisher bandwidth and AI company compute for no incremental training value, which is exactly the inefficiency Pay Per Use is meant to price away.

AI crawlers already generate more than 50 billion requests to the Cloudflare network every day, just under 1 percent of all web requests the company sees. That figure is a small slice of total traffic, but it concentrates on the exact pages publishers care about most, the news, reference, and product content that trains frontier models. The Pay Per Crawl marketplace, announced in July 2025, gave domain owners the option to set a flat per-request price across their site. The new Pay Per Use model shifts the price to the value the content produces inside an AI system, such as a search answer or a cited source.

For publishers, the logic is straightforward: if a chatbot answers a user query using their article, they should be paid for the value delivered. For AI companies, the logic is harder. A retrieval-augmented model may pull the same article hundreds of times to verify a single answer, which makes a per-fetch fee punishing and a per-use fee workable only if the AI vendor can audit its own usage.

Google’s Counteroffer and the Wider Backlash

Cloudflare’s announcement singles out the world’s largest search engine, a thinly veiled reference to Google, as having access to about “2x more information” than other AI companies because it is hard for sites to stay visible in Google Search without also being used for AI training. Google has pushed back on that framing, pointing to Google Extended, a separate bot that lets publishers opt out of training and AI products like Gemini Apps and Vertex API without losing their place in Google Search.

The flagship Googlebot crawler still pulls content for Search and for AI features such as AI Overviews and AI Mode, and that dual use is exactly what Cloudflare now wants labelled and priced. Other search and AI vendors, from OpenAI to Anthropic to Perplexity, face the same September 15 reckoning if they keep mixing search, agent, and training crawlers under one identity. The new BotBase tooling, released to enterprise customers on Cloudflare’s second Content Independence Day on July 1, 2026, is meant to let publishers tell those bots apart at a glance.

What Defenders and Data Collectors Do Now

For sites running on Cloudflare, the practical work is small but real. Owners of ad-supported properties need to decide whether to keep Cloudflare’s new defaults, which block mixed-use crawlers by default, or to override them and allow specific bots. Owners of free-tier sites get the new defaults automatically and may not realise their content is now invisible to a class of crawlers they never opted into blocking. For larger enterprises, Bot Management offers finer-grained controls, including the ability to classify traffic as Search, Agent, or Training.

For the data teams on the other side of the trade, the message is the same as it has been for three years: every protected site is an arms race. A scraper that worked last quarter can break this quarter because a target site updated its bot detection, its fingerprinting model, or its CDN rules. The teams still running collection in-house increasingly find themselves maintaining anti-bot circumvention instead of analysing the data they set out to collect. Managed scraping services have grown for the same reason: a vendor whose full-time job is staying ahead of the detection arms race looks very different from a brittle in-house script that breaks the moment a target site ships an update.

Frequently Asked Questions

What is anti-crawler protection?

Anti-crawler protection is the layer of tools a website uses to tell automated visitors apart from humans and decide which bots get through. It covers challenge-response tests such as CAPTCHAs, behavioural analysis of mouse movement and click timing, device and browser fingerprinting, machine learning risk scoring, and access pattern monitoring of IP reputation and request rates. The simplest version is the three-second browser check visitors see when they first land on a protected page.

Why is Cloudflare blocking AI crawlers in September 2026?

Cloudflare announced on July 1, 2026 that starting September 15 its default settings would block mixed-use crawlers, bots that combine search, agent use, and training in one identity, from any site that hosts ads. The policy targets the AI training pipeline directly, where publishers say their content has been harvested without payment. The defaults apply to new Cloudflare customers, new sites for existing customers, and all existing free customers.

How much of the internet is now automated traffic?

Automated traffic reached 51 percent of all web requests in 2024, the first year bots outnumbered humans on the open web, according to the Imperva Bad Bot Report. Bad bots alone accounted for 37 percent of all traffic that year, up from 32 percent the year before. By 2026, automated traffic had climbed further, to more than 53 percent of all web requests.

What is Cloudflare’s Pay Per Crawl programme?

Pay Per Crawl is a Cloudflare marketplace launched in 2025 that lets website owners set a flat per-request price for AI crawlers, allow free access, or block bots entirely. The programme is now evolving into Pay Per Use, which charges AI companies when content produces value inside an AI system, such as appearing in a search answer, rather than per page fetched. Cloudflare’s initial Pay Per Use partners are Ceramic.ai and You.com.

What is Cloudflare’s AI Labyrinth?

AI Labyrinth is an opt-in Cloudflare feature announced on March 19, 2025 that serves AI-generated decoy pages to crawlers it identifies as unauthorised. The pages look realistic enough to lure a bot deep into a maze of irrelevant content, wasting its compute budget. Any visitor that follows four hidden links deep is almost certainly a bot, which gives Cloudflare fresh signal to feed back into its detection models.

Will these changes affect Google search results?

Cloudflare’s September 15 defaults do not change how Googlebot indexes sites for traditional search. The change targets crawlers that blend search with AI training and agent use under a single identity. Google separately offers Google Extended, a bot that lets publishers opt out of training and AI products such as Gemini Apps and Vertex API without losing visibility in Google Search itself.