Berlin, August 5 – Websites are increasingly facing challenges from aggressive scraping by artificial intelligence (AI) companies, leading to downtime and restricted access for legitimate users. In response to this growing issue, a new system named Anubis has been deployed, utilizing a Proof-of-Work (PoW) scheme to mitigate these threats.
Anubis: A Compromise for Website Protection
Anubis, developed by Techaro, aims to strike a balance between protecting websites from malicious scraping and maintaining accessibility for human users. The system operates on a principle similar to Hashcash, a proposed PoW scheme designed to reduce email spam. The core idea is that the additional computational load imposed by Anubis is negligible for individual users but becomes substantial and costly for mass scrapers.
According to developers, this approach makes aggressive data collection economically unfeasible for AI companies, thereby reducing the strain on server resources and preventing service disruptions. The system is currently running Anubis version v1.26.2.
How Anubis Works: The Proof-of-Work Mechanism
When a website protected by Anubis detects suspicious activity, it presents a challenge page requiring a small computational effort from the user’s browser. This Proof-of-Work is designed to be quick and imperceptible for human visitors but accumulates significantly for automated bots attempting to scrape large volumes of data. This mechanism effectively filters out AI scrapers without hindering regular website traffic.
The administrator of a website utilizing Anubis stated, “You are seeing this because the administrator of this website has set up Anubis to protect the server against the scourge of AI companies aggressively scraping websites. This can and does cause downtime for the websites, which makes their resources inaccessible for everyone.”
Addressing the AI Scraping Challenge
The rise of AI companies aggressively scraping websites has altered the traditional social contract of web hosting. This practice often leads to server overloads, making resources inaccessible and causing significant downtime. Anubis provides a temporary solution, allowing website administrators more time to develop advanced fingerprinting techniques for identifying headless browsers-such as those used by AI bots-based on characteristics like font rendering.
The developers acknowledge that Anubis currently requires modern JavaScript features to function correctly. “Sadly, you must enable JavaScript to get past this challenge. This is required because AI companies have changed the social contract around how website hosting works. A no-JS solution is a work-in-progress,” they explained. Users with plugins like JShelter may need to disable them for domains protected by Anubis to ensure seamless access.
Future Developments and User Impact
While Anubis offers an immediate defense against AI scraping, its developers are actively working on a no-JavaScript solution to further enhance accessibility and user experience. The current implementation is a placeholder, designed to buy time for the development of more sophisticated identification methods that can distinguish between legitimate users and automated scrapers without requiring client-side JavaScript execution.
The system is part of a broader effort by Techaro, a company based in Canada, to provide robust web protection solutions. The mascot design for Anubis was created by CELPHASE.
This initiative underscores the ongoing battle between website administrators and AI companies over data access and server integrity, highlighting the need for innovative solutions to maintain a fair and accessible internet for all.
Source: https://www.hu-berlin.de/en/open-humboldt/latest-news/details/prof-schmidt-awarded-the-2026-humboldt-preis
