The Web’s War on AI Bots Is Making Humans Prove They’re Human

A countdown ticks from three to zero. A line about spam bots flashes past. An IP address stares back at the visitor before the real page even loads. That interstitial, once a rare annoyance, now guards a growing slice of the internet, and new traffic data explains why it keeps showing up.

Automated requests passed human ones on Cloudflare’s network for the first time in early June, a crossover the company’s own chief executive had not expected until next year. Cloudflare’s answer is a September 15 deadline forcing AI crawlers to declare their purpose or get shut out. The bots are the target. The friction is landing on everyone else.

Bots Now Outnumber Humans on the Open Web

Cloudflare’s its live bot-versus-human traffic split put automated requests at 57.5% of all HTML traffic on its network as of early June, based on figures its co-founder and chief executive, Matthew Prince, shared publicly. Humans made up the remaining 42.5%, the first time machines have outnumbered people since the web began keeping this kind of score.

A separate yardstick points the same direction from a different angle. The Thales 2026 Bad Bot Report, the security vendor’s thirteenth annual study, found bots driving 53% of 2025’s web traffic, up from 51% the year before. The two numbers measure different baskets. Cloudflare counts raw HTML requests on its own network; Thales pools data across a wider client base that weighs attack traffic more heavily. Both still land on the same side of fifty percent.

Cybersecurity firm HUMAN Security added its own data point in March, reporting that AI agent traffic alone grew nearly 8,000% year over year in 2025. Its researchers describe agentic browsing, software completing tasks on a person’s behalf, as a newly distinct category sitting alongside the older split of good bots and bad ones.

Cloudflare Draws a Line for September 15

Cloudflare routes something like one in five of the world’s websites, a scale that makes its default settings a policy lever few single companies get to pull. Starting September 15, any newly onboarded site and every customer still on the free tier will automatically block crawlers that blend search indexing with AI training or live agent fetching, on any page that carries ads.

The company frames the change as new defaults blocking Training and Agent crawlers while leaving Search crawling untouched, since an ad on a page signals a website owner wanted a person, not a scraper, to land there.

Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.

Matthew Prince, Cloudflare’s co-founder and chief executive, said in the announcement of the policy.

Cloudflare sorts crawlers into three buckets rather than a single allow-or-block switch:

• Search – crawlers that index a page to answer questions about it later, the traditional search-engine bargain of crawl now, refer traffic later.
• Agent – automated activity acting in real time on a person’s behalf, including chat fetch bots and browser-use style agents completing a task.
• Training – crawlers pulling content to train or fine-tune a model, whether or not that same bot also does Search or Agent work.

Google, Microsoft’s Bing and Apple all run crawlers that blend more than one of those purposes, which is exactly what put them in the new policy’s path. Each offers a separate opt-out bot that can keep its main crawler clear of the block, though site owners have to actively choose it. The settings involved, and what changes for a publisher who does nothing before the cutoff, are the subject of Cloudflare’s September 15 policy reset.

Why Does My Browser Keep Asking Me to Prove I’m Human?

Verification screens are multiplying because the old tests stopped working. Picking crosswalks and traffic lights out of a photo grid used to stump machines; modern AI systems now solve those puzzles at high success rates, so sites are quietly shifting to checks a visitor never sees: browser fingerprinting, behavioral scoring, and increasingly, a gesture read straight off a webcam.

Google is piloting exactly that inside its Cloud Fraud Defense product. The test asks a visitor to move a hand in front of their camera, then relies on a 21-point knuckle scan for liveness to judge whether the motion came from a real person. Google says the clip processes in real time, is never linked to an identity, and is deleted once the check ends. The company avoids calling it biometric authentication, though pulling anatomical landmarks off a live camera feed sits in a regulatory space few privacy rules were written for.

Long before any puzzle or gesture appears, most checkpoints have already scored the visit on IP reputation, timezone match against declared location, and whether the reported hardware makes sense together, the layer of checks detailed in what that browser check actually verifies. Fail enough of those silently, and the visible puzzle never even has to run.

The 73,000-to-1 Problem

Cloudflare put a number on the imbalance last year that still anchors the whole policy fight: for every reader a crawler sends back to a publisher, some AI crawlers took tens of thousands of pages first.

Anthropic’s crawler, in other words, visited a page roughly 73,000 times for every reader it sent back over the course of 2025, the widest gap in Cloudflare’s published figures. That asymmetry is the economic argument behind Pay Per Crawl, the tollbooth Cloudflare opened in private beta back in July 2025 and is now rebuilding into a model called Pay Per Use, paying publishers when their content actually surfaces inside an AI answer rather than merely when a bot fetches it. Ceramic.ai and You.com are the first two AI companies wired into the new version.

The scraping fight has already reached the courts. HUMAN Security’s own tracking counts more than 70 copyright suits against AI firms, including the New York Times and Chicago Tribune’s case against Perplexity and a first-round ruling that fourteen publishers won against the Canadian AI startup Cohere.

Europe’s Courts and Publishers Push Back

The pressure is not only American. Technology.org separately noted that a German court had held Google liable over false claims generated inside its AI Overviews, one flashpoint in what the outlet called a content war escalating on every front. European publishers are not responding with one playbook either.

Politico’s own product leadership told trade outlet Press Gazette that Politico.eu had been redesigned to be as legible to AI crawlers as possible, a bet on visibility rather than the block-everything approach many American publishers are taking.

• 20% of the world’s web traffic runs through Cloudflare’s network, by the Columbia Journalism Review’s count, which is why one company’s default settings carry so much weight.
• 5.6 million websites had added OpenAI’s GPTBot to their disallow lists by late 2025, per web-metrics firm BuiltWith’s data, up from 3.3 million in July, a jump of nearly 70%, with 5.6 million sites now disallow GPTBot and Anthropic’s ClaudeBot blocked on 5.8 million more.
• 336% was the year-over-year rise in sites blocking AI crawlers outright, according to Tollbit, a company that helps publishers charge crawlers for access instead.

Not every European or American outlet is choosing the same side of that split, which is precisely why the next twelve months look less like a single rulebook settling in and more like a live experiment running in public.

Real Users Absorb the Friction Machines Created

The bot side of this fight is not standing still either. A wave of stealth browser tools now markets itself on defeating exactly the systems Cloudflare, Akamai and DataDome run, patching Chromium at the operating-system level so automated traffic looks indistinguishable from a person clicking through a Tuesday afternoon. One such vendor describes antibot systems as able to detect far more traffic than they currently block, holding back only because flagging real customers by mistake costs more than letting some bots through.

That tolerance is exactly what is expected to shrink. As bot traffic keeps climbing, the same vendor argues, the calculation flips: letting bots through starts costing more than occasionally blocking a human, and detection tools sitting quietly on monitor-only settings get switched to block. Every tightened threshold lands on the same shared queue of visitors, bots and people alike, waiting for a page to decide what they are.

The checkpoint that opened this piece will reset in three seconds, ask its question again tomorrow, and keep asking it of a species of visitor it was never quite built to sort cleanly.

Frequently Asked Questions

Why did Cloudflare set a September deadline instead of blocking crawlers immediately?

Cloudflare is phasing the change through its default settings rather than an instant network-wide switch, applying the new block only to newly onboarded sites, new sites from existing customers, and free-tier accounts starting September 15. Paying customers who have already customized their own crawler rules will not see the new defaults applied automatically, which limits how much of the web changes overnight.

Does this change affect whether my site shows up in Google search results?

No. Cloudflare’s Search category remains allowed by default even after September 15. The complication is that Google’s main crawler, Googlebot, also feeds AI Overviews and AI Mode, so a site blocking Training traffic can end up blocking parts of Google’s AI features unless it separately opts in through Google’s dedicated Google-Extended crawler.

Why do I keep seeing the same verification screen on completely unrelated websites?

Because one infrastructure company sits in front of a large share of the sites you visit. Cloudflare alone routes roughly a fifth of global web traffic, so the identical widget, and the identical three-second wait, shows up again and again across sites that otherwise have nothing to do with each other. The recurring pattern behind that repetition is covered in why the same check keeps recurring.

Can a real person get mistaken for a bot?

Yes, and it already happens more than most users realize. Browser automation company Browser Use notes that antibot systems currently set their blocking thresholds conservatively on purpose, since wrongly flagging a paying customer costs a business more than letting a stray bot through. As bot volume keeps rising, that cushion is expected to shrink, meaning more borderline human sessions could get challenged or blocked outright.

What happened to Cloudflare’s Pay Per Crawl marketplace?

It is evolving rather than disappearing. Pay Per Crawl launched in private beta in July 2025, letting publishers charge AI companies per fetch. Cloudflare is now rebuilding it into Pay Per Use, which pays publishers when their content actually appears inside an AI-generated answer, with Ceramic.ai and You.com as the first two partners live on the new model.

Do these verification tools collect biometric data?

Some are starting to. Google’s hand-gesture reCAPTCHA test extracts 21 knuckle-point coordinates from a short camera clip to judge liveness, then deletes the footage once the check finishes, according to Google’s own privacy disclosures. The company does not classify it as biometric authentication, but capturing anatomical landmarks from a live camera feed still sits in a legal grey zone that predates this kind of check.