Berlin, Germany – In response to the growing challenge of aggressive AI companies scraping websites, a new protection system named Anubis has been deployed. This system, currently running version v1.27.0, aims to safeguard server stability and ensure that website resources remain accessible for all users by implementing a Proof-of-Work scheme.
The Challenge of AI Scraping
Website administrators have increasingly faced downtime and resource inaccessibility due to AI companies aggressively scraping content. This practice, while beneficial for AI development, places a significant burden on website servers, leading to service disruptions for legitimate human users.
Anubis: A Compromise Solution
Anubis is presented as a compromise solution to this problem. It utilizes a Proof-of-Work scheme, similar to Hashcash, which was initially proposed to reduce email spam. The core idea behind Anubis is to introduce a minimal, ignorable load for individual users, while making mass scraping significantly more expensive and resource-intensive for AI operations.
This approach aims to deter large-scale scraping activities without unduly inconveniencing typical website visitors. The system is designed to add up the computational cost for scrapers, making their operations less economically viable.
Technical Requirements and Future Developments
Anubis requires the use of modern JavaScript features to function effectively. This means that users with plugins like JShelter, which disable certain JavaScript functionalities, will need to disable them for domains protected by Anubis to bypass the challenge. The developers acknowledge that this is a temporary solution, and a no-JavaScript alternative is currently in progress.
The current implementation is seen as a placeholder while more advanced methods for fingerprinting and identifying headless browsers (e.g., through font rendering analysis) are developed. These future improvements aim to eliminate the need for presenting Proof-of-Work challenges to users who are clearly legitimate.
The Impact on Web Hosting and Accessibility
The deployment of Anubis highlights a shift in the social contract of website hosting, driven by the actions of AI companies. The need for such protective measures underscores the ongoing tension between data accessibility for AI training and the operational stability of web services.
The system is developed by Techaro, with mascot design by CELPHASE, and includes standard legal information such as Impressum, Datenschutzerklärung, and Barrierefreiheit.
