In a move aimed at empowering content creators and combating a growing concern, website security giant Cloudflare unveiled a new tool on Wednesday. This tool allows web hosting customers to block AI bots from scraping their website content.
These bots often collect data without permission, using it to train machine learning models. Cloudflare’s decision to offer this blocking feature stems from customer frustration with AI bots and a desire to “help preserve a safe internet for content creators,” according to a company statement.
Here is what the statement says:
We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly. To help, we’ve added a brand new one-click to block all AI bots.
Website owners already have a basic defense against unwanted bots – the robots.txt file. Placed at the root of a website, this file instructs automated crawlers (including some AI bots) on which areas they’re not allowed to access.
However, the issue of AI content scraping is more complex. Some believe the entire field of generative AI is built on stolen data, leading to a rise in lawsuits targeting AI companies. In response, some content-scraping firms have offered website owners a (somewhat ironic) “opt-out” option, allowing them to (theoretically) prevent their content from being used to train AI models.
Last year, OpenAI, aware of these worries, released instructions on how to block its GPTbot crawler with a robots.txt file. Google followed suit with similar measures a month later.
Cloudflare joined the fight in September 2023, offering website owners a tool to block compliant AI bots. The response was significant, with 85% of Cloudflare’s customers reportedly enabling this blocking feature.
