Cloudflare’s September 15 AI Crawler Block Reshapes the Web

Cloudflare said Wednesday that it will block mixed-use AI crawlers from ad-supported pages by default on September 15, 2026, as part of its second annual Content Independence Day. The deadline turns a year-old “pay or be scraped” stance into a structural rule applied at the CDN layer, and it shifts who decides what AI systems can read on the open web.

The default flip applies to new Cloudflare customers, new sites set up by existing customers, and all existing free-tier customers who have not adjusted their settings before that date. For publishers, sysadmins, and AI companies, it is the moment when AI traffic stops being background noise and becomes infrastructure policy.

September 15 Becomes the New Default for Ad Pages

Starting September 15, 2026, Cloudflare will block mixed-use crawlers from any page that hosts ads by default. A “mixed-use” crawler is one that does more than one thing on a site: it indexes for search, retrieves pages for an AI agent, and pulls content for model training, all under a single bot identity or a tightly coupled system.

Cloudflare is defining what counts as acceptable crawler behavior on the open web. Search, AI agent activity, and model training become separate categories that site owners can treat differently, and a crawler that blends them will face the most restrictive rule that applies to any of its purposes on a given page.

The changes apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers, the company said. The new defaults are live in the Cloudflare dashboard, where customers can review and change them before the September 15 deadline.

How Cloudflare Now Sorts Bots

Cloudflare now classifies AI-related crawlers into three buckets. Search crawlers index a site to answer questions later and are tied to referral traffic. Agent crawlers act for a person in real time, such as ChatGPT-User or browser agents like Gemini or Claude operating Chrome. Training crawlers pull content to train or fine-tune a model.

On ad-supported pages under the new defaults, Search stays allowed. Training and Agent crawlers are blocked by default for new customers, new sites, and free-tier customers who have not changed their settings. Bot operators should run separate crawlers for each behavior, the company said, so websites can see why a bot is visiting and decide whether to allow or block it.

Mixed-Use Crawlers Are the Real Target

The structural change is the combined-crawler rule. Cloudflare will treat multi-purpose crawlers based on their overall behavior, applying the strictest rule that applies. A crawler that performs both Search and Training will be blocked if a site blocks Training. Cloudflare names Googlebot, Applebot, and Bingbot as examples, since each crawls for both search and AI training.

The argument is about consent as much as blocking. A publisher might reasonably want to appear in ordinary search results, refuse to have its archive used to train a future model, and allow real-time AI search summaries but reject bulk ingestion for training. Those are distinct business decisions, and mixed-use crawling collapses them into a single yes-or-no gate.

The technical demand is simple in theory: run separate crawlers, label them clearly, and let websites choose.

The political demand is sharper, to stop using search dependency as a bargaining chip to obtain AI permissions publishers would not grant if the choices were cleanly separated. The detailed breakdown sits in the breakdown of how the rule can block Googlebot.

Google Is the Elephant in the Room

Cloudflare’s announcement refers to the “world’s largest search engine” without needing to say the name. The company specifically calls out that search engine as having access to about “2x more information” than other AI companies because it makes it difficult for customers to remain discoverable without being used for AI. The full context sits in the September 15 default-block policy in full.

Google’s defense is that publishers already have a control. Google Extended lets site owners opt out of having content used for training and AI products and services like Gemini Apps and Vertex API. Its use does not affect a site’s inclusion in Google Search. The tech giant’s flagship Googlebot still crawls for Search, including AI features like AI Overviews and AI Mode.

That is where Cloudflare’s critique lands. If a publisher wants to block the main Googlebot to stop training, it risks damaging visibility in conventional search. The practical choice is messier than the product documentation makes it sound. The same crawler identity now supports a blue-link index, a generated answer, a shopping assistant, and a model training pipeline. Platform companies call this integration. Publishers experience it as coercion by architecture.

Free-Tier Site Operators Are the Silent Stakeholders

The default-setting change will land unevenly across Cloudflare customers. For paid customers who already manage their settings, September 15 is a date to review the dashboard. For the long tail of free-tier customers, it is a date when settings may flip under their feet if no one logs in.

The exposure is bigger than it sounds. Most organizations do not have a single authoritative map of their Cloudflare footprint, with marketing owning the main site, product owning docs, support owning a knowledge base, engineering owning developer pages, and somewhere a long-retired agency owning a forgotten microsite still sitting behind Cloudflare on a free plan. When defaults become policy, the policy lands on whoever owns the forgotten microsite.

A security-minded admin may welcome stricter blocking, while an SEO team may panic when AI visibility drops. A developer relations team may want documentation reachable by AI coding assistants, while legal may want training blocked. The safe assumption in the default treats the entire estate as if it needed the most conservative setting.

Different pages carry different commercial logic. A public press release, a paid research report, a troubleshooting forum, and a product documentation page do not have the same answer to the same crawler question. The September 15 default treats them all the same way unless the owner adjusts the dashboard in advance.

• More than 50% of crawl traffic from good bots goes to re-fetching pages that have not changed, per Cloudflare’s network data.
• More than 50% of traffic online is now non-human, Cloudflare CEO Matthew Prince said in the announcement.
• AI training accounts for the majority of crawler requests on Cloudflare’s network, up from roughly 20% in spring 2025.
• Daily AI agent requests increased by more than 1,700% over the year on Cloudflare’s network.
• 8% click-through rate when Google shows an AI summary, versus about half that without, per a 2025 Pew Research Center study.

From Pay Per Crawl to Pay Per Use

Last year Cloudflare launched Pay Per Crawl, letting site owners charge AI companies for scraping pages. The company is now reshaping it into Pay Per Use, which ties payment to the value content creates inside an AI product rather than to the raw HTTP request. A single page might be crawled once and cited in thousands of answers, or crawled repeatedly and never used at all. Creators want to be paid fairly for the value they provide, Cloudflare said. The newer model is detailed in the new AI traffic options Cloudflare just launched.

The first partners are Ceramic.ai and You.com. Ceramic has built a pay-per-query model so publishers who opt in can be paid when their content appears in Ceramic’s search results. You.com lets agents pay on demand for a specific piece of premium content with no upfront commitment. Cloudflare’s role is the infrastructure layer that lets these models scale across its network.

To scale the future of AI search, we need a partner with massive reach and a shared commitment to transparency and fair compensation. Cloudflare allows us to easily and programmatically scale our operations. By bringing our pay-per-query model to their network, we ensure millions of content owners can seamlessly opt in to be compensated every single time their content appears in our search results.

The pull quote came from Anna Patterson, founder and CEO of Ceramic.ai, in the Cloudflare announcement post.

Cloudflare is also launching a research programme to use its network signals to steer answer engines toward fresh content, an experiment the company says is limited to search and excludes foundation model training. The programme is described in the research programme to make AI search smarter.

What Site Owners Should Do Before September 15

For most Cloudflare customers, the September 15 default change is a review task that does not require crisis management. The defaults are live in the dashboard now. The question is whether the current settings reflect the policy the organization actually wants for each property, or only the property the marketing team remembers to check.

Cloudflare’s move gives administrators more levers, but it also makes crawler governance a real operational discipline that now lives somewhere on the org chart. The framework’s success hinges on whether major operators differentiate their bots by behavior, since the alternative is a compromise between blocking AI training and maintaining search visibility.

1. Audit every domain on Cloudflare, including forgotten microsites and old agency properties still on the free tier.
2. Decide which bot category each property should allow, block, or negotiate, page by page where it matters.
3. Separate search crawlers from training and agent crawlers where possible, since the strictest rule will apply to any bot that blends them.
4. Track the new Pay Per Use experiments with Ceramic.ai and You.com if monetization is part of the answer.

Frequently Asked Questions

What is Cloudflare’s mixed-use AI crawler block?

Cloudflare announced on July 1, 2026 that it will block mixed-use AI crawlers from ad-supported pages by default starting September 15, 2026. A mixed-use crawler performs search indexing, agent activity, and model training under a single identity or a tightly coupled system.

Which Cloudflare customers does the September 15 default change affect?

The new defaults apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers. Paid customers who already manage their settings can review or change them in the dashboard at any time before September 15.

Will the new Cloudflare defaults block Googlebot?

Not by default. Search crawlers, including Googlebot, stay allowed on ad-supported pages under the new defaults. If a site enables the older “Block AI bots” setting, the combined-crawler rule can extend that block to mixed-use bots like Googlebot, Applebot, and Bingbot, since each crawls for both search and AI training.

What is Pay Per Use and how is it different from Pay Per Crawl?

Pay Per Crawl, launched in 2025, let publishers charge AI companies for crawling their pages. Pay Per Use is the newer model that ties payment to the value content creates inside an AI product, such as being cited in an answer, rather than to the raw request. Initial partners are Ceramic.ai and You.com.

Why is Cloudflare making this change now?

Cloudflare’s announcement cites the milestone that bots now account for the majority of web traffic, a shift not expected until next year. Cloudflare data also shows more than 50% of crawl traffic from good bots goes to re-fetching pages that have not changed, and AI training now accounts for the majority of crawler requests on the network.