Cloudflare Unveils New Feature to Block Bots & AI Crawlers in a Single Click

Cloudflare introduced a new feature that lets users block AI bots and web scrapers with just one click.

This new tool, available to all Cloudflare customers, including those on the free tier, aims to safeguard original content from unauthorized scraping and use by AI companies.

The rise of generative AI has led to an increased demand for data to train large language models (LLMs).

This has resulted in a surge of AI bots scraping content from websites, often without the consent of the content creators.

While some AI companies, such as Google and OpenAI, have made efforts to be transparent about their scraping activities, many others have not.

This has raised concerns among website operators about the unauthorized use of their content.

One-Click Solution for All Users

Cloudflare’s new feature simplifies blocking unwanted AI bots. Users can enable this protection by navigating to the Security > Bots section of the Cloudflare dashboard and toggling the “AI Scrapers and Crawlers” option.

This feature will automatically update to include new bot fingerprints as they are identified, ensuring continuous protection against evolving scraping techniques.

Cloudflare’s analysis of its network traffic revealed that AI bots like Bytespider, Amazonbot, ClaudeBot, and GPTBot are among the most active in scraping content.

Bytespider, operated by ByteDance, leads in both the number of requests and the extent of its crawling activities. GPTBot, managed by OpenAI, follows closely behind.

These bots are used to gather data for training AI models that power popular applications like ChatGPT and Alexa.

Cloudflare’s machine learning models play a crucial role in identifying and blocking AI bots, even when they attempt to disguise themselves as legitimate web browsers.

The company’s global network, which processes over 57 million requests per second, provides a robust foundation for detecting and mitigating bot activity.

By leveraging this extensive data, Cloudflare can promptly identify new scraping tools and adapt its defenses accordingly.

In addition to the one-click blocking feature, Cloudflare has introduced tools for reporting misbehaving AI bots.

Enterprise Bot Management customers can submit reports via Bot Analytics, while all users can use a dedicated reporting tool to flag unauthorized scraping activities. This proactive approach ensures that Cloudflare protects its users’ content.

Cloudflare’s new feature represents a significant step towards preserving the integrity of online content in the face of increasing AI-driven scraping.

Cloudflare empowers website operators to maintain control over their content and protect it from unauthorized use by providing an easy-to-use tool for blocking AI bots.

As AI technology continues to evolve, Cloudflare remains committed to enhancing its defenses and supporting content creators in their efforts to safeguard their intellectual property.