Cloudflare Will Block AI Crawlers by Default From September 15

Cloudflare said it will block AI crawlers from ad-supported web pages by default starting September 15, 2026, splitting automated traffic into three buckets called Search, Agent, and Training and forcing every site behind its network to choose which machines get through. The change lands as automated bots have crossed 53% of all web traffic, according to Imperva’s 2026 Bad Bot Report, and as publishers have spent a year losing both their content and the readers who used to come with it.

The shift comes after AI crawlers have scraped publisher content at scale while delivering almost no referral traffic back. Cloudflare’s Content Independence Day announcement, dated July 1, 2026, frames the move as a response to that imbalance. “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” Cloudflare co-founder and CEO Matthew Prince said. The default change will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers, the company said.

The Three-Way Split for Bots

For most of the web’s history, the word “bot” meant a search engine crawler. Cloudflare’s new taxonomy drops that assumption and asks three separate questions about every automated request: what it is doing on the site, what it is storing, and how it will reshare the content. A crawler that returns referral traffic by answering questions later is Search. A system that fetches a page in real time to complete a task for a human waiting on the other end is Agent. A crawler that absorbs the content into a model’s weights is Training.

Cloudflare’s blog post frames this as a transparency push. Bots that combine Search with Training should split themselves into separate crawlers, and multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be evaluated against all of their purposes, not just the friendliest one. The strictest applicable rule wins, the post said, which means a site that blocks Training will block a Search bot that also trains. That detail is the structural heart of the policy, and it explains why Cloudflare is willing to put the world’s largest search engines in the same bucket as smaller AI scrapers.

Cloudflare separately launched a feature called BotBase that lists every known crawler the network tracks and tags each one with one or more of the three classifications. The company is also testing a new robots.txt signal called use, which lets a site declare whether a bot can store, reference, or fully reproduce its content.

The Numbers That Forced the Switch

Two 2026 traffic studies arrived within weeks of each other and gave Cloudflare the numbers it needed to justify a default switch. Imperva’s 2026 Bad Bot Report, titled “Bots in the Agentic Age,” puts the headline plainly: automated bot traffic accounted for more than 53% of all web traffic in 2025, up from 51% the year before, with human activity falling to 47%. The report frames the shift as structural, not a spike driven by any single attack, and notes that AI agents no longer just scan sites but retrieve data, execute workflows, and act on behalf of users.

• 53% of web traffic in 2025 came from bots (Imperva)
• 27% of bot attacks targeted API endpoints, not user-facing pages (Imperva)
• AI bot activity rose 300% in 2025 (Akamai)
• Publishing accounted for 40% of all AI bot traffic (Akamai)
• AI chatbots drove about 96% less referral traffic than traditional Google search in Q4 2024 (Akamai)

Akamai’s State of the Internet report, released the same month, adds the publishing angle: AI bot activity surged by 300% in 2025, and publishing organizations made up 40% of all AI bot traffic, the highest concentration of any industry it tracked. OpenAI generated the highest single share of AI bot traffic to media companies, the report found. AI training crawlers made up 63% of all AI bots targeting the media industry, with 37% of that traffic aimed at publishers specifically. AI fetchers, the bots that pull a page in real time to answer a user’s question, represented 24% of AI bot activity against media and concentrated 43% of it on publishing.

Pay Per Crawl Evolves Into Pay Per Use

Cloudflare launched its Pay Per Crawl marketplace a year ago as a way for publishers to charge AI bots per request. That model now evolves into Pay Per Use, which pays publishers when their content actually creates value, not when it is fetched. The change addresses a long-standing complaint from publishers: more than 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages, Cloudflare’s own data shows, which inflates infrastructure costs without delivering readers.

Two partners are live at launch. Ceramic.ai pays publishers when their content appears inside Ceramic’s AI search results. You.com pays when it accesses a publisher’s premium content. Other AI companies can build their own payment flows on top of Cloudflare’s marketplace, the company said. Stack Overflow became the first Cloudflare customer to commercialize its data through Pay Per Crawl in February 2026, charging AI bots per request through the marketplace.

Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent.

The quote comes from Prince’s announcement, and it points to a second change Cloudflare made public: it has quietly stopped distinguishing “AI bots” from other bots and now classifies them by behavior, since most modern automation touches more than one task. A crawler that fetches a search index, drives an agent, and trains a model must identify all three roles, Cloudflare said, or face the strictest combined rule.

Google’s Counter-Move in the UK

Google has spent the past year resisting the framing that its crawlers should be classed alongside other AI bots. The company offers Google Extended, a separate bot that lets site owners opt out of having their content used to train Gemini Apps and Vertex API, and Googlebot continues to crawl the web for Search, including the AI Overviews and AI Mode features that now sit at the top of the results page. Cloudflare’s announcement pointedly noted that one major search engine enjoys access to “about 2x more information” than its rivals, because site owners cannot easily stay discoverable without being used for AI.

The UK’s Competition and Markets Authority has stepped into the same fight from a different angle. Under the UK CMA opt-out framework, online publishers can opt out of appearing in Google’s AI Overviews without losing their place in standard search results. “It is crucial that content publishers, including news organisations, have appropriate bargaining power over how their content is used,” CMA Chief Executive Sarah Cardell said. The CMA described the rules as a “world-first requirement.” Google told the BBC it was “engaging with regulators to ensure website owners have the right tools as user preferences evolve.”

What Still Hasn’t Been Solved

Cloudflare’s announcement addresses the crawlers it can see. Several of the harder problems sit outside its reach. The rule change applies only to sites behind Cloudflare’s network, the company said. Sites on other infrastructure will need to make their own calls about how to classify AI traffic. The threats that worry security teams most often do not look like crawlers at all.

The Imperva report also flags a quieter risk. Twenty-seven percent of bot attacks in 2025 targeted API endpoints rather than user-facing pages, allowing attackers to bypass front-end protections and operate at machine speed. Financial services bore 24% of all bot attacks and 46% of account takeover incidents, the report found, a sign that the worst abuse is moving off the public web and into back-end systems. Cloudflare’s new taxonomy is, in effect, a perimeter policy for a threat that increasingly lives inside the building.

Akamai’s report adds a publishing-specific number that helps frame the cost of the policy shift. AI chatbots drove about 96% less referral traffic than traditional Google search in Q4 2024, Akamai found, sharply reducing a critical source of audience and revenue. As the September 15 anti-crawler reset pushes publishers toward opting in to AI bots selectively, the open question is which sites can afford to block the most crawlers and which still need the referrers they get from search.

• Self-hosted crawlers that do not identify themselves
• Per-crawler detection that fails on shared hosting ranges
• Scrapers that switch user-agents on every request
• API attacks that target back-end systems directly
• AI agents that route traffic through residential proxies

Patrick Sullivan, Akamai’s Chief Technology Officer for Security Strategy, framed the publisher problem plainly: “AI bots are eroding core revenue streams, such as advertising and subscriptions, while driving up infrastructure costs and diminishing brand visibility.” The September 15 deadline puts a calendar on those losses, but it does not, on its own, replace the audience that has already gone.

Frequently Asked Questions

When does Cloudflare start blocking AI crawlers by default?

Cloudflare said the new defaults take effect on September 15, 2026. They apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers, the company said.

What is the Search, Agent, and Training split?

It is Cloudflare’s new taxonomy for AI crawlers. Search crawlers index content to answer questions later. Agent crawlers fetch pages in real time to complete tasks for a user. Training crawlers absorb content to train or fine-tune models. Multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be evaluated against all of their purposes, not just the friendliest one.

What is Pay Per Use?

Pay Per Use is the evolution of Cloudflare’s Pay Per Crawl marketplace. Instead of paying publishers per fetch, the new model pays publishers when an AI company’s content actually creates value, such as when a publisher’s article is cited inside an AI answer.

Which AI companies are participating in Pay Per Use?

Cloudflare launched Pay Per Use with two partners: Ceramic.ai and You.com. Other AI companies can build custom payment flows on top of the marketplace, the company said. Stack Overflow became the first customer on the earlier Pay Per Crawl program in February 2026.

Does the new Cloudflare policy affect Google search rankings?

No. Sites that block Training bots in Cloudflare’s new defaults will still be crawled by Googlebot for standard Search, including AI Overviews and AI Mode, because Googlebot is classified primarily as Search. Google separately offers Google Extended, which lets publishers opt out of training Gemini on their content. The UK CMA has also given publishers a separate route to opt out of Google’s AI Overviews without losing their place in regular search.

For more on what the browser-level experience looks like when these checks fire, see what happens during an anti-crawler browser check.