Cloudflare Creates AI Crawler Tollbooth
Matthew Prince (Hacker News, Slashdot):
The problem is whether you create content to sell ads, sell subscriptions, or just to know that people value what you’ve created, an AI-driven web doesn’t reward content creators the way that the old search-driven web did. And that means the deal that Google made to take content in exchange for sending you traffic just doesn’t make sense anymore.
Instead of being a fair trade, the web is being stripmined by AI crawlers with content creators seeing almost no traffic and therefore almost no value.
That changes today, July 1, what we’re calling Content Independence Day. Cloudflare, along with a majority of the world’s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content. That content is the fuel that powers AI engines, and so it’s only fair that content creators are compensated directly for it.
This is pretty cool, but we’re also dangerously close to Cloudflare basically being the whole internet.
In a separate post, Cloudflare’s David Belson, head of data insight, and Sam Rhea, VP of product, published data illustrating the disparity between what AI crawlers take and the referral traffic they send back to websites.
During the period between June 19 and 26, 2025, for example, “Anthropic’s AI platform Claude made nearly 71,000 HTML page requests for every HTML page referral,” observe Belson and Rhea. We must note that these measures only track traffic from the Claude website, not the app, as the app does not emit a
Referer:header. The same goes for the other AI vendors.
I’m concerned that this default goes too far. Cloudflare has enormous power to intercept web traffic, because they’ve effectively re-centralized DNS for so many websites. While Matthew’s reasons for doing this are good, it should still be an opt-in feature. The open web should by default be open.
[…]
Cloudflare has a series of blog posts today with more details. In one post, they outline how AI crawlers can use HTTP Signatures (similar to what ActivityPub uses) to identify themselves if they have a relationship with Cloudflare for making payments to web publishers. When enabled, Cloudflare will return an HTTP 402 “payment required” response. There’s a mechanism for crawlers to say how much they will pay or to accept the listed price.
[…]
I can also imagine a harmless bot accidentally getting mislabelled as an AI crawler. Cloudflare has significant control even though they aren’t even the ones hosting your web site. According to a companion press release today, Cloudflare proxies traffic for 20% of the web.
Previously:
- MacInTouch Paused
- Please Stop Externalizing Your Costs Directly Into My Face
- Microsoft’s Suleyman on AI Scraping
- AI Companies Ignoring Robots.txt
Update (2025-07-04): Vladimir Prelovac:
Cloudflare launched pay per crawl service in an attempt to centralize control of AI crawling economy.
Interestingly there is an open source effort by Coinbase which may be a better way to achieve this for publishers (and this could be the first actually useful thing to come out of the crypto world). This banks on existing http 402 response spec and is conveniently called x402.org
Now if only it wasn’t so darn hard to setup a wallet for your grandma in her browser so we could have decent micropayments on the web. Something I think about a lot in the context of Kagi/Orion.
Update (2025-07-08): Adam Engst:
There are undoubtedly numerous concerns with pay-per-crawl, not the least of which is that it would put Cloudflare in a position of even greater power within the Internet ecosystem. It could also hinder academic research and open source projects that lack substantial funding.
However, what I find even more interesting about pay-per-crawl is how it might revive HTTP response code 402 as a more general method of enabling direct transactions between producers and consumers. We’re getting close to some of the micropayment-related ideas in Ted Nelson’s largely theoretical Project Xanadu, which could radically democratize commerce on the Internet (I’ve been beating this drum for decades; see “Xanadu Light,” 29 November 1993).
5 Comments RSS · Twitter · Mastodon
Open Web choices are:
- Geoblock most of the world and throttle everything hard.
- Use megacorps to handle problems caused mostly by other megacorps.
Why don’t web hosts or DNS providers get in on this? If it takes off, I’m sure there will be some copycats
Obviously, this isn’t a *real* solution so of course there are tons of problems with it. The real solution is legislation that forces AI companies to honor website’s choices to allow crawling or not, but doesn’t disable search traffic
Cloudflare's outsized control over much of the internet should be a much bigger concern for everybody.
AI crawlers were monetizing content they didn't create, so Cloudflare, who also did not create said content, started monetizing it. and the people cheered
@Someone Spot on. Depressing, isn't it?
@Vintner +1. Full disclosure, I use Cloudflare *exclusively for DNS* on public resources; I use its reverse proxy only through Zero Trust / Teams and tunnels and only for private resources. I pay for none of this, but I want to acknowledge that technically-speaking CF does get a lot right.
@Manx Nah, the *real* solution is to take a long, hard look at copyright, and ask whether it's still serving us at all well. AI is a symptom both of a real desire for access to information and a massive concentration of power in the hands of a few (which Western governments are now nakedly kowtowing too, incidentally, rather making a mockery of their maximalist positions on copyright), but micropayments have been a proposed solution for access to digital information long before now. Copyright is pointless if not enforceable, actively does harm when it restricts information in a digital age (and especially to disadvantaged groups in society), and doesn't benefit creators living in a world run by undemocratic AI companies. We've got to start thinking big. But between pearl-clutching on the one hand and brute force on the other, of course we're not about to start now. :(
