Cloudflare Cuts Off Free AI Access to Content

For nearly 25 years, a silent agreement defined the internet: content creators allowed companies like Google to crawl their websites in exchange for traffic. This traffic powered advertising, subscriptions, and online growth. But with the rise of AI, this deal has fallen apart.

On July 1, Cloudflare, one of the internet’s key infrastructure providers, announced it would block AI crawlers from accessing websites on its platform unless they pay content creators. CEO Matthew Prince called it “Content Independence Day.”

AI tools such as ChatGPT and Google’s AI Overviews increasingly bypass traditional search by answering user queries directly, drawing from scraped web content while driving little or no traffic to sources. According to Cloudflare, OpenAI generates 750 times less traffic than Google, while Anthropic is even lower, 30,000 times less.

“Instead of a fair trade, the web is being strip-mined by AI crawlers,” Prince wrote. Content creators provide the fuel for AI systems but are left out of the resulting value chain.

A New Business Model for Content Access

Cloudflare controls access to about 20% of all websites globally. Leveraging this scale, the company now plans to create a marketplace where AI companies can license content directly from creators. The model would reward content not for virality, but for its utility in filling AI knowledge gaps—a system metaphorically described as “filling holes in a block of Swiss cheese.”

This marks the first time an internet infrastructure company has changed its default behavior to block AI access. While OpenAI and other firms have made paid content deals with platforms like Reddit and the Financial Times, Cloudflare’s shift is broader and systemic.

Cloudflare claims this model could usher in a “new golden age of content creation,” where quality and depth matter more than traffic volume. But this vision raises difficult questions: Who defines valuable content? How is compensation determined? Could this fragment the internet even further?