Cloudflare defaults to blocking AI crawlers and launches a paid crawling mechanism.
The world's largest web security service provider Cloudflare announced that starting from July 2, new domains will default to blocking AI crawlers unless explicit permission is obtained or paid crawling is arranged.
Cloudflare calls this measure 'Content Independence Day,' which has received support from over 1 million Cloudflare customers and 35% of the top 1,000 websites worldwide, including internationally renowned media such as the Associated Press, Time Magazine, The Atlantic, BuzzFeed, Reddit, Quora, and Universal Music Group.
According to a report by The Verge, Cloudflare has also launched a 'Pay Per Crawl' mechanism, allowing publishers to set prices for AI crawlers. AI companies can view pricing plans and decide whether to subscribe to the paid service or abandon crawling.
Currently, this paid crawling service is only available to select top online media and content creators, but Cloudflare plans to expand the system's scale.
Further Reading:
Google traffic avalanche! AI summaries seem convenient but are a killer for content websites, how does the official respond?
Cloudflare has been fighting AI crawlers for a long time: content websites have been exploited too much by OpenAI.
Cloudflare is a web infrastructure company that provides website hosting and security services. After the rise of generative AI led to a large number of AI crawlers being used to train models, they are actively considering solutions.
In 2023, Cloudflare began assisting website owners in combating AI crawlers, initially only blocking crawlers that complied with the robots.txt file. By 2024, it can allow websites to block all AI bots, regardless of whether they comply with the site's robots.txt file.
Cloudflare CEO Matthew Prince pointed out that ten years ago, Google would bring one visit to content websites for every two pages crawled, but now Google takes away 18 pages for you to get just one visitor.
After generative AI companies entered the scene, this ratio has become even more extreme. Six months ago, the ratio of OpenAI crawling web pages to visitors browsing web pages was 250:1, now it is 1,500:1; while Anthropic was 6,000 to 1 six months ago, now it is 60,000:1.
Further Reading:
AI Taiwan) Jian Lifeng: The knowledge economy is declining! What is the next wave of business opportunities for AI?
The community praises Cloudflare for blocking AI crawlers, but are research purposes also being mistakenly blocked?
On social media, Cloudflare's new measures have received widespread praise.
Former Stability AI audio VP Ed Newton-Rex stated, 'This is the path forward.' Anonymous trader Romano RNR praised, 'Cloudflare solves problems again,' while former Google employee and current SEO consultant Pedro Dias also believes that a force is needed to restore the balance between content websites and crawlers.
Stack Overflow CEO Prashanth Chandrasekar stated that community platforms providing fuel for large language models (LLMs) should also receive compensation for their contributions so they can reinvest back into the community.
However, some experts have expressed concerns. (MIT Technology Review) reported that MIT Media Lab doctoral candidate Shayne Longpre warned that defaulting to block AI crawlers may interfere with non-commercial uses, such as research.
He stated that not all AI systems are competing with online media or have commercial purposes; personal use and open research should not be sacrificed because of this.
'OpenAI is in trouble! Cloudflare opens the door to block AI crawlers, no longer afraid of content being exploited?' This article was first published in 'Crypto City.'