{"id":48321,"date":"2025-07-02T16:17:58","date_gmt":"2025-07-02T20:17:58","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=48321"},"modified":"2025-07-08T15:41:55","modified_gmt":"2025-07-08T19:41:55","slug":"cloudflare-creates-ai-crawler-tollbooth","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2025\/07\/02\/cloudflare-creates-ai-crawler-tollbooth\/","title":{"rendered":"Cloudflare Creates AI Crawler Tollbooth"},"content":{"rendered":"

Matthew Prince<\/a> (Hacker News<\/a>, Slashdot<\/a>):<\/p>\n

The problem is whether you create content to sell ads, sell subscriptions, or just to know that people value what you’ve created, an AI-driven web doesn’t reward content creators the way that the old search-driven web did. And that means the deal that Google made to take content in exchange for sending you traffic just doesn’t make sense anymore.<\/p>

Instead of being a fair trade, the web is being stripmined by AI crawlers with content creators seeing almost no traffic and therefore almost no value.<\/p>

That changes today, July 1, what we’re calling Content Independence Day. Cloudflare, along with a majority of the world’s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content. That content is the fuel that powers AI engines, and so it’s only fair that content creators are compensated directly for it.<\/p><\/blockquote>\n\n

thaddeus<\/a>:<\/p>\n

This is pretty cool, but we’re also dangerously close to Cloudflare basically being the whole internet.<\/p><\/blockquote>\n\n

Thomas Claburn<\/a>:<\/p>\n

In a separate post<\/a>, Cloudflare’s David Belson, head of data insight, and Sam Rhea, VP of product, published data illustrating the disparity between what AI crawlers take and the referral traffic they send back to websites.<\/p>

During the period between June 19 and 26, 2025, for example, “Anthropic’s AI platform Claude made nearly 71,000 HTML page requests for every HTML page referral,” observe Belson and Rhea. We must note that these measures only track traffic from the Claude website, not the app, as the app does not emit a Referer:<\/code> header. The same goes for the other AI vendors.<\/p><\/blockquote>\n\n

Manton Reece<\/a>:<\/p>\n

\n

I’m concerned that this default goes too far. Cloudflare has enormous power to intercept web traffic, because they’ve effectively re-centralized DNS for so many websites. While Matthew’s reasons for doing this are good, it should still be an opt-in feature. The open web should by default be open.<\/p>\n

[…]<\/p>\n

Cloudflare has a series of blog posts today with more details. In one post<\/a>, they outline how AI crawlers can use HTTP Signatures (similar to what ActivityPub uses) to identify themselves if they have a relationship with Cloudflare for making payments to web publishers. When enabled, Cloudflare will return an HTTP 402 “payment required” response. There’s a mechanism for crawlers to say how much they will pay or to accept the listed price.<\/p>\n

[…]<\/p>\n

I can also imagine a harmless bot accidentally getting mislabelled as an AI crawler. Cloudflare has significant control even though they aren’t even the ones hosting your web site<\/em>. According to a companion press release today<\/a>, Cloudflare proxies traffic for 20% of the web.<\/p>\n<\/blockquote>\n\n

Previously:<\/p>\n