Berlin, Germany – In response to the growing threat of aggressive AI companies scraping websites, a new bot protection system named Anubis, developed by Techaro, is being widely deployed. This system aims to protect server resources and prevent downtime caused by intensive scraping activities, ensuring accessibility for all users.
Anubis: A Compromise for Website Protection
Anubis operates on a Proof-of-Work (PoW) scheme, similar to Hashcash, which was originally proposed to reduce email spam. The core idea behind Anubis is to introduce a negligible additional load for individual users, while making mass scraping significantly more expensive and resource-intensive for AI companies. This approach serves as a compromise to mitigate the impact of aggressive data harvesting.
How Anubis Protects Websites
The system works by presenting a challenge that requires a small amount of computational effort to solve. For a single user, this process is quick and goes largely unnoticed. However, for large-scale scrapers attempting to access vast amounts of data, the cumulative computational burden becomes substantial, effectively deterring malicious activity.
A spokesperson for Anubis stated, “At individual scales the additional load is ignorable, but at mass scraper levels it adds up and makes scraping much more expensive.”
Addressing Modern Web Challenges
The deployment of Anubis highlights a significant shift in the social contract surrounding website hosting, driven by the practices of AI companies. These practices have often led to server strain and downtime, affecting legitimate users.
Technical Requirements and Future Developments
Anubis requires modern JavaScript features to function correctly. Users with browser plugins like JShelter that disable JavaScript may need to temporarily disable them for domains protected by Anubis. The developers acknowledge this current limitation and are actively working on a no-JavaScript solution.
“Sadly, you must enable JavaScript to get past this challenge. This is required because AI companies have changed the social contract around how website hosting works. A no-JS solution is a work-in-progress,” the developers noted.
This is considered a placeholder solution while more advanced methods, such as fingerprinting and identifying headless browsers based on font rendering, are developed. These future advancements aim to present the Proof-of-Work challenge only to those users who are more likely to be illegitimate scrapers.
Version Information and Development Team
The current version of Anubis running on protected websites is v1.26.0. The system is developed by Techaro, with the mascot design created by CELPHASE. The initiative is a testament to the ongoing efforts to secure online resources in an increasingly automated digital landscape.
