Internet infrastructure giant Cloudflare has announced that it will now block AI bots from scraping website content by default, in a move designed to return control to content creators while balancing the needs of AI companies.
The new policy, unveiled Tuesday, means that any new website hosted on Cloudflare’s vast content delivery network (CDN) will automatically prevent AI crawlers from accessing its data — unless website owners explicitly grant permission.
This builds on a feature Cloudflare first introduced in September 2023, which allowed publishers to block AI crawlers with a single click. Now, blocking will be the default setting for all new Cloudflare sites.
Cloudflare CEO Matthew Prince said the decision aims to “rebalance control in favor of content creators” and safeguard the sustainability of the open web.
“AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators while still helping AI companies innovate,” Prince said. “This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.”
As part of the shift, Cloudflare is also introducing a new ‘pay-per-crawl’ model that enables publishers to charge AI companies for access to their website content. The idea is to give content owners an alternative revenue stream while ensuring AI companies can still obtain training data ethically and transparently.
The move could have wide-reaching implications: Cloudflare’s CDN handles a large share of global web traffic, meaning AI companies that rely on scraping the open web for training large language models (LLMs) may now face additional hurdles or costs.
Industry observers say the announcement signals a broader push for more equitable arrangements between tech platforms, creators, and AI developers, as the debate over fair data use intensifies.
