Cloudflare has started to block AI crawlers by default to protect content publishers' control on the Internet.
MAIN CONTENT
Cloudflare implements a default mechanism to block AI crawlers to protect publishers' data.
The Pay per Crawl program helps AI and publishers negotiate compensation when accessing data.
Many large AI developers like OpenAI oppose the new policy, sparking debate over access rights and data collection frequency.
What has Cloudflare done to control AI bots' access to websites?
Cloudflare officially applies a default mechanism to block AI crawlers for new domains, aiming to return control to content publishers.
This move is an expansion of the tools Cloudflare previously introduced, such as the one-click AI bot blocking feature and a dashboard to monitor data collection activities. According to CEO Matthew Prince, the goal is to balance protecting content ownership with supporting AI development.
AI crawlers have collected content without limits. We want to return control to creators while allowing AI companies to continue innovating.
Matthew Prince, CEO of Cloudflare, July 2025
Unlike traditional CDN services that improve web access speed, Cloudflare now requires website owners to determine whether AI bots can access data or completely block access.
How does the Pay per Crawl program operate and what benefits does it offer to publishers?
Pay per Crawl is an intermediary marketplace operated by Cloudflare, allowing publishers and AI companies to negotiate fees when AI bots collect data.
Both parties need to register accounts on Cloudflare to set up agreements, ensuring publishers' rights through revenue from granting data access, while also clarifying the source and purpose of AI crawlers.
Why do some AI developers oppose Cloudflare's new policy?
OpenAI and some other companies refuse to participate in the program, opposing Cloudflare becoming an intermediary between publishers and AI development.
OpenAI asserts that it always complies with the robots.txt file to respect the website's choice regarding allowing crawlers access. However, an analysis by Cloudflare shows that OpenAI's data collection rate far exceeds the referral traffic: about 17,000 crawls per actual visit, compared to Google at 14 times.
AI crawlers have put significant pressure on websites and negatively affected user experience. If Cloudflare's system works effectively, it helps limit the large-scale data collection capabilities of AI bots.
Matthew Holman, technology lawyer, 2025, CNBC
What is the response of publishers and the importance of data control?
Many major media companies such as TIME, The Associated Press, Conde Nast, The Atlantic, ADWEEK, and Fortune commit to blocking AI bots by default.
While traditionally, publishers accepted Google's data collection in exchange for traffic and advertising revenue, AI platforms currently do not provide similar interaction or economic benefits, leading to a need to protect original content online.
Cloudflare also aims to collaborate with AI developers so that crawlers must publicly identify themselves and their purposes.
Original content is what makes the Internet one of the greatest inventions of the last century. We must work together to protect it.
Matthew Prince, CEO of Cloudflare, 2025
Frequently Asked Questions
What impact does blocking AI crawlers have on regular users? Blocking AI crawlers does not affect the experience of normal users accessing the site, but only controls automated data collection bots. What does Pay per Crawl help publishers receive? The program helps publishers earn revenue from allowing AI bots to access data, ensuring content control. Why does OpenAI oppose Cloudflare's control mechanism? OpenAI believes that Cloudflare creates unnecessary intermediaries and that they always comply with robots.txt to respect websites. Does the new policy reduce AI development capabilities? Limiting access to large data sets may reduce free training data, forcing AI to adhere to clearer regulations. Do major publishers agree with Cloudflare? Most large media companies have accepted and participated in the content protection campaign initiated by Cloudflare.
Source: https://tintucbitcoin.com/cong-ty-ai-phan-doi-cloudflare-cam-bot/
Thank you for reading this article!
Please Like, Comment, and Follow TinTucBitcoin to stay updated with the latest news about the cryptocurrency market and not miss any important information!
