Blocking the Bots by Default
Cloudflare is changing the default rules of the web it sits in front of. Starting September 15, 2026, the company will block mixed use crawlers by default on pages that display ads, a change that applies to new customers, new sites from existing customers, and all free tier users. The target is a specific and growing menace: crawlers that blend search indexing, AI agent activity, and model training into a single bot, harvesting content for AI systems under cover of behaving like a traditional search engine.
The move inverts a decades old assumption. For most of the web's history, being crawled was good, the price of admission to search traffic and therefore to an audience. That bargain is breaking. When a crawler takes your content to train a model or answer a query directly, it does not send you a visitor in return, it substitutes for the visit. Cloudflare is betting that publishers, facing that broken exchange, would rather block by default and negotiate access than continue giving their work away to systems that route around them.
When Most Traffic Is Not Human
Cloudflare's co founder and chief executive, Matthew Prince, put the rationale in stark terms, saying that now that the majority of traffic on the internet is non human, the company must go further and act faster so that a sustainable ecosystem can emerge. That single statistic, that most internet traffic is now bots rather than people, is the premise for everything that follows. A web economy built on the assumption of human eyeballs cannot function when the visitors are overwhelmingly machines that never see an ad or click a link.
The company also points out that more than half of AI crawler traffic involves re fetching pages that have not changed, wasting publisher bandwidth and compute for no benefit to the site. Prince framed the default changes as an attempt to encourage mixed use crawlers to separate search from agent use and training, so that publishers can make informed choices about each. The underlying argument is that the current free for all is not just unfair to publishers, it is technically wasteful and ultimately unsustainable for everyone.
From Pay Per Crawl to Pay Per Use
Blocking is only half the strategy. Cloudflare is evolving its Pay Per Crawl marketplace, which lets sites charge AI bots to scrape them, into a broader Pay Per Use model that lets publishers charge AI companies when their content actually creates value rather than merely when it is fetched. That distinction is subtle and important. Charging per crawl prices the act of access, while charging per use attempts to price the value the content delivers when it informs an answer or a model, which is closer to what the content is actually worth.
The most ambitious piece is the Monetization Gateway, which will let a site charge for any resource behind Cloudflare, a web page, a dataset, an API, or even a tool exposed over the Model Context Protocol, with charges settling in stablecoins over the open x402 protocol and no payment stack for the publisher to build. Early partners include Ceramic.ai and You.com. If it works, Cloudflare is not just defending publishers, it is constructing the toll infrastructure for a machine to machine web where value moves in tiny automated payments.
The Bid to Become Infrastructure
Read together, these moves are a bid for Cloudflare to become the economic layer of the AI web, the place where access is negotiated and value is settled between the entities that produce content and the AI systems that consume it. Cloudflare is uniquely positioned to attempt this because it already sits in front of a huge fraction of the internet, seeing the traffic, classifying the bots, and holding the technical leverage to block or meter them. Payment rails are only credible if you also control the gate, and Cloudflare controls a great many gates.
That position is powerful and, for some, uncomfortable. A single company deciding the default terms on which AI can access the web, and building the marketplace where that access is priced, concentrates enormous influence over the flow of information and money online. Cloudflare frames itself as acting on behalf of publishers against extractive AI companies, and there is truth in that. But the same infrastructure that protects content creators also positions Cloudflare as a toll collector on the AI economy, a role that will attract scrutiny as it grows.
The Publishers' Dilemma
For publishers, Cloudflare's tools are a lifeline and a gamble at once. The lifeline is obvious: a mechanism to reclaim value from AI systems that have been consuming their work for free and, increasingly, answering users' questions directly so the visit never happens. For a media industry battered by exactly that dynamic, the ability to block or charge machine access is the first real leverage it has been handed in years, and many will grab it. The gamble is subtler and concerns what happens to reach.
Blocking crawlers protects content but also risks invisibility in the AI answer engines that a growing share of users now rely on to find information. A publisher that walls off its content entirely may preserve its rights while vanishing from the surfaces where audiences increasingly form. The strategic task becomes deciding which crawlers to allow, at what price, and for what purpose, a genuinely difficult optimization that Cloudflare's classification tools are meant to enable. The technology gives publishers choices they lacked, but it does not make the choices easy, and getting them wrong cuts in both directions.
What Builders Should Take Away
For engineers and product leaders, this is a signal to design for a web where crawler access is negotiated and metered rather than assumed. If you build systems that depend on freely scraping the open web, the ground is shifting under you, and the era of unlimited, unpriced access is ending. Conversely, if you produce valuable content or data, the tooling to charge for machine access is arriving, and ignoring it means continuing to subsidize the AI systems that consume your work.
The deeper shift is architectural. A web that settles micropayments between bots over open protocols, that exposes datasets and even MCP tools as paid resources, is a materially different platform to build on than the free and open web most engineers grew up with. Whether Cloudflare specifically becomes its economic layer is less certain than the direction itself. The unpriced, human centric web is giving way to a metered, machine centric one, and the teams that understand the new economics early will build the businesses that thrive in it.
