How to combat large amounts of Ai scrapers

I've seen people suggesting and using Anubis, haven't used it myself though.

I especially love the irony of Anubis using yesterday's hype thing to combat today's.
i tried Anubis and it works great the only issue is it wont support multiple subdomains
Second Anubis, just finished by setup yesterday i have it of a oracle cloud frre tier vps, which depending on the domain routes the traffic to services hosted on the vps itself or to my server ar home. Relatively easy to setup, blocks most requests with very few false positives (one of which for example it would aggressively challenge by thunderbird trying to reach my baikal instance). I set a bit more aggresive rules than default (i also block googlebot and bingbot, since i received a bit more requests than I'd like). In like 10 hours it straight up denied about 5000 requests from the ai-catchall ruleset (mostly amazonbot) and challenged about 10000, mostly from a block of IPs in singapore, some of the hosts having the user agent of a Macintosh with PowerPC. They all sure love to explore the public repos on my git server.
I'm in the process of changing servers for an upgrade, the old one still hosting more services while I setup the new one. The old one now does run audibly quiter. I don't even want to think how much electricity went wasted because of those bots
- You probably don't need me to tell you, but keep good backups. Friend of mine recently had his account nuked without any reason given, and without the possibility of recourse.
I’ve had trouble with it using a vpn and privacy browsers. It often blocks me until I use a default browser.
I have a dumb question.. what is preventing the crawlers from just eating the shit and just burn though the energy to get through the computational task?
- It'll still slow them down and reduce load on your server. I also think many of these crawlers focus on volume; time spent computing the hash is time not spent crawling someone else's site.
I was going to recommend that, very easy to setup

You need yo block the alibaba subnets primarily. In my experience this is where most of them originate

Nephentes that shit. Poison every scraper until they start respecting robot.txt. Purposefully use llm.txt to trap the fuckers.

this is really awesome but i use a vps so i dont know how they would feel about this being deployed
- You most likely pay for a maximum CPU capacity and your Server cant go above that no matter what you run. Your vps provider doesn't care what that CPU power is used for.
  However with limited resources, the tarpit might use up all CPU power and the rest of the webserver will crawl to a halt.

Wern't there a few AI maze projects in the works? I wonder if running one of those for a bit will cause you to be added to an ignore list, clearly they dont respect your robots file.

Tar pits I think is the term they use to pollute AI data.

If nginx, here's an open-source blocker/honeypot: https://github.com/raminf/RoboNope-nginx

If you have it set up to be proxied or hosted by Cloudflare, they have their own solution: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/

I wonder why that RoboNope doesn't just make a fail2ban entry for anything that accesses a disallowed url and drop them entirely.
Actually this look like it would do something similiar, then dumps them to fail2ban after the re-access the honeypot page too many times: https://petermolnar.net/article/anti-ai-nepenthes-fail2ban/
ill check robonope out seems promising

does anubis not work?

i can only get it to protect one container. i have 3 that i need protected and i cant figure out how to run more then one instance of it.

If you're able to, use GeoIP ranges to only allow access from the countries you want.

That immediately limits a lot of everything

Then - again if you're able to - use a block list that covers known scrapers in case they're in your country.

I use pfBlockerNG on my pfSense firewall for exactly this.

In my case I use https://www.bunkerweb.io/ as my proxy for that, but there are other tools like for example https://github.com/TecharoHQ/anubis

bunkerweb looks intresting

I've seen people mention Anubis, the other one I heard about in a blog post that's maybe worth looking into is go-away.

How do you know it's "AI" scrappers?

I've have my server up before AI was a thing.

It's totally normal to get thousands of bot hits and to get scraped.

I use crowdsec to mitigate it. But you will always get bot hits.

Some of them are at least honest and have it as a user agent.
- Is ignoring robots.txt considered "honest"?
bot hits i dont care my issue is when i see the same ip querying every file on 3 resource intensive sites millions of times
- Do you have a proper robots.txt file?
  Do they do weird things like invalid url, invalid post tries? Weird user agents?
  Millions of times by the same ip sound much more like vulnerability proving than crawler.
  If that's the case fail to ban or crowdsec. Should be easy to set up a rule to ban an inhumane number of hits per second on certain resources.

crowdsec is arguably not completely open source, but I'm very satisfied with it.

The scraper blocklist on crowdsec requires a paid subscription, though, or did you find another workaround?
- I don't remember how I set it up a long time ago. But when I look at my server logs I only see myself.
  Afaik I just added the biggests lists. But I don't remember.

Well, someone had great idea to use zipbombs. I saw it somewhere but I don’t remember where.

Anubis has this built in if it detects bots it turns the diffuclty to impossible

What's bothering you?

Is it to give out data for AI training? I guess you can't fundamentally protect against this, except by limiting how much content is provided to each address.
Or is it the resource strain that it causes on your server? In that case i recommend limiting how much a single client / IP address can request in a day.

its the strain of it i mostly run instances and frontends so the training is not a huge problem
- the keyword you need is "DDoS protection" i guess
  it keeps the server from getting overloaded due to too many requests

I don't have opensource solutions, but CloudFlare had some news about a system that I didn't read about (saw two headlines) last week. Dunno if it works or not.