Anti-Crawler Protection and Cloudflare’s September 15 Reset

The three-second pause at the top of many websites, the one that flashes the line “Anti-Crawler Protection is checking your browser and IP” before any HTML loads, has moved from a niche security extra to a default front door for the open web. Most readers ignore it; behind it, every page request is being judged against a stack of fingerprint, network, and behavioural checks, and the rules behind those checks are being rewritten.

Cloudflare, the network in front of more than 20% of all websites, said on July 1, 2026 that its default behaviour for AI crawlers changes on September 15, 2026. New domains joining the platform will block Training and Agent bots on any page that displays ads, leaving Search bots allowed. Multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked wherever their operators have not split search and training into separate user agents. The same shift lands for existing customers as their security settings come up for review, and the verification page most people skip today is the front edge of that change.

The Three-Second Check Becomes the Front Door

The anti-crawler message is the lightest tier of a layered system. A walkthrough of how the anti-crawler browser check works walks through every stage from TLS handshake to behaviour scoring, including the five detection layers described in the next section of this article.

Why the check now appears on most large sites comes down to traffic mix. Bots accounted for more than 53% of all web traffic in 2025, up from 51% in 2024, according to the 2026 Imperva Bad Bot Report findings. The same report puts financial services as the heaviest-hit sector at 24% of all bot attacks and 46% of account takeover incidents in 2025. Anti-bot tooling has migrated from a security add-on to a core production layer, sitting beside hosting, storage, and CDN capacity, which is why a site with modest human readership can still see most of its requests arrive from automated scripts.

What Cloudflare Changes on September 15

The update is a taxonomy and a default. Cloudflare’s updated AI crawler policy post introduces three new labels (Search, Agent, and Training) and lets every customer block or allow each one independently, including on the company’s Free tier. The new default applies on any page that displays ads, blocking Training and Agent crawlers there while leaving Search allowed by default.

On pages that carry advertising, the new default blocks Training and Agent crawlers and leaves Search alone. The most restrictive applicable rule wins for any given page, which is the new policy’s core mechanism. Search bots feed referrals and ad impressions back to the publisher; Training and Agent bots usually do not, which is why the default tilts against them on monetised pages. Cloudflare laid out the ad-signal rationale directly in its announcement.

Now that the majority of traffic on the internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.

That line is from Matthew Prince, Cloudflare’s co-founder and CEO, in the announcement introducing Cloudflare’s September 15 default flip. The shift sits between two prior products Cloudflare has shipped: last year’s one-click “Block AI Bots” option, and the Pay-Per-Crawl marketplace that lets publishers charge AI companies per page fetched. The new default is the third step in a program to put publishers back in control of who reads their pages. A fourth product, a new robots.txt extension called the content-use signal, lets publishers declare whether content may be stored, excerpted, or reproduced at all.

Multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be evaluated by every classification they touch, which means a bot that crawls for both search and training will be blocked wherever a customer has set Training to block. BotBase, the new dashboard listing every verified bot with its Search, Agent, and Training classification, ships at the same time and lets customers write precise allow or block rules without writing detection code. The three classifications are each controlled through the same Security settings panel as the legacy Block AI Bots option, so customers do not need new tooling to manage them. Existing customers can opt out of the new defaults at any time through Security settings, with reminders going out ahead of the date.

How the Check Decides in Under a Second

The judgment happens in two stages, and most visitors will never see either one. Before any HTML returns, the server runs a TLS handshake check, looks up the IP against ASN ranges and proxy pools, and inspects the order, casing, and HTTP/2 pseudo-header priority of the incoming request. If that early bundle looks clean, a JavaScript payload runs in the visitor’s browser and reports back on canvas rendering, WebGL output, hardware sensors, and a few dozen other API calls. A session that agrees with itself at every layer is allowed through. A session that contradicts itself anywhere is slowed, challenged, or blocked.

1. TLS fingerprinting reads the cipher suite order, TLS extensions, and GREASE values that differ between real browsers and scripted clients.
2. IP reputation compares the address against ASN data, known data centre ranges, and residential proxy pools.
3. HTTP header ordering checks the name order, casing, and HTTP/2 pseudo-header priority against the claimed browser.
4. JavaScript fingerprinting reads browser APIs, canvas and WebGL output, and hardware sensor data inside the visitor’s browser.
5. Behaviour biometrics tracks mouse path, scroll velocity, and typing cadence across the session, not just at the door.

Each layer on its own is unreliable. A user-agent string can be copied in seconds. Residential proxies make any IP look like a home connection. Pairing a Chrome user-agent string with a Python TLS fingerprint is enough for every major vendor on the market to flag the request, and the five layers exist because none of them is decisive on its own.

The Crawl-to-Referral Numbers Behind the Flip

The growth rate is what pushed operators to flip the default. AI crawler traffic rose 757% in 2024, the fastest of any category Cloudflare tracked, and 50 billion AI crawler requests now pass through the network every day.

The query response share tells the rest of the story. Only 2.2% of AI bot requests respond to a real user query; the remaining 97.8% is data extraction with no return visit, no ad impression, and no referral back to the source site. That imbalance between high crawl volume and outbound referrals is what publishers cite in their push for default blocks at the network level. The growth rate accelerated even as ad-supported publishers demanded a share of the value, and the September 15 default is Cloudflare’s response to a multi-year negotiation that has produced little so far for content creators. Crawlers for AI training now account for 49.9% of all AI bot traffic on the network, the largest category by share.

The crawl-to-referral ratios put numbers on the imbalance, and the gap is widest for the crawlers publishers cannot do without. Anthropic’s ClaudeBot crawls 20,583 pages for every single referral it sends back to a publisher. OpenAI’s GPTBot crawls 1,255 pages per referral, which is a tighter ratio but still leaves the publisher subsidising the crawler’s bandwidth costs. Meta’s crawler sends zero referrals back to source publishers, with every page it touches feeding Llama training and Meta AI products with no reciprocal value to content creators.

Even blocking a single bot is incomplete on its own. Cloudflare’s own measurement, published alongside the policy update, found 70.6% of websites that actively block the ChatGPT-User agent still appear in AI-generated citations, because the underlying model was trained on data collected before the block took effect. The lesson from that measurement is that one-off opt-outs do not undo past training, which is why the September 15 default is set at the network layer rather than left to individual site owners.

Who Pays, Who Gains, and What Changes Next

For publishers, the new policy brings a sharper toolset. The September 15 anti-crawler reset sits between two earlier Cloudflare moves: last year’s one-click “Block AI Scrapers and Crawlers” control, and the Pay-Per-Crawl marketplace that lets publishers charge AI companies per page fetched. The new BotBase dashboard lists every verified bot with its Search, Agent, and Training classification, so a site owner can write precise allow or block rules without maintaining custom detection code. Pay-Per-Crawl is evolving into Pay Per Use, a model that pays publishers based on the value their content generates inside an AI product rather than the number of pages fetched, with Ceramic.ai and You.com as launch partners.

For AI labs, the immediate pressure is to split their crawlers. Cloudflare’s call is for operators to maintain separate user agents for Search, Agent, and Training, so a publisher who wants indexing but not training extraction can allow one and block the other. Under Cloudflare’s default-block policy for AI crawlers going live September 15, operators that do not split face the most restrictive rule across all of their behaviours. Anthropic, OpenAI, Google, Apple, and Microsoft will each have to decide before September 15 whether to maintain separate crawlers for each role, with implications for every publisher that runs ads.

For scrapers and data teams, the picture is more crowded. Industry tracking puts the global web scraping market at $1.17 billion in 2026, on track to reach $2.28 billion by 2030. The major anti-bot vendors, including Cloudflare, DataDome, PerimeterX, Kasada, Akamai, Incapsula, F5, and AWS WAF, each score requests differently, so a scraper tuned for one will fail against another within days of any model update. A growing slice of that spend goes into staying ahead of detection stacks that update faster than in-house code can be redeployed.

Frequently Asked Questions

What is anti-crawler protection?

Anti-crawler protection is the layered system a website uses to tell automated requests from real visitors. It runs a TLS fingerprint check, looks the IP address up against reputation databases, orders and inspects HTTP headers, executes JavaScript inside the visitor’s browser, and tracks behaviour across the session. A clean passage through every layer releases the request in a few seconds; an automated request is slowed, challenged, or blocked at one of the layers, usually with a verification page that says the system is checking the browser and IP.

Why does the anti-crawler check keep popping up?

Bots accounted for more than 53% of all web traffic in 2025, up from 51% in 2024, according to the Imperva 2026 Bad Bot Report. With the majority of requests now coming from automated scripts, websites run anti-crawler checks as a default rather than only when an attack is suspected. The same report puts 24% of all bot attacks on financial services and flags APIs as the dominant new target for scripted reconnaissance.

Will Cloudflare’s September 15 change stop me from reading sites?

No. The change targets AI training crawlers and AI agent bots, not human visitors or search engines. Search bots such as Googlebot remain allowed by default on pages that show ads, and the policy does not add a new challenge for human browsers reading content. Anyone visiting a Cloudflare-protected site with a normal web browser should see no change in how a page loads on or after September 15.

How does a publisher turn off the new defaults?

Cloudflare customers can opt out of the September 15 changes through their Security settings at any time before the date, with reminders going out ahead of it. The opt-out keeps existing behaviours for Training crawlers, including the multi-purpose crawlers such as Googlebot, Applebot, and BingBot that would otherwise be blocked wherever Training is set to block. Bot Management customers can also flip individual Search, Agent, and Training classifications independently.

Can AI companies pay to crawl instead of being blocked?

Yes. Cloudflare’s Pay-Per-Crawl marketplace has been live for a year and lets publishers charge AI operators per page fetched. The follow-on model, Pay Per Use, pays publishers based on the value their content generates inside an AI product, with Ceramic.ai and You.com as launch partners. A separate robots.txt extension called the content-use signal lets a publisher declare whether content may be stored, excerpted, or reproduced, which sets the default for compliant AI crawlers.

Disclaimer: This article is for informational purposes and covers general web security and platform policy. Anti-bot policies and detection stacks change frequently; consult Cloudflare’s published documentation for current settings. Figures cited are accurate as of publication in July 2026.