Ai scraping is an effective DDoS on the entire interent
Ai scraping is an effective DDoS on the entire interent
Excerpt from a message I just posted in a #diaspora team internal forum category. The context here is that I recently get pinged by slowness/load spikes on the diaspora* project web infrastructure (Discourse, Wiki, the project website, ...), and looking at the traffic logs makes me impressively angr...
Count them as ad visits, to make big tech pay for better hardware or line?
That opens you up to getting accused of click fraud, as AdNauseam found out the hard way but its worth it if you can squeeze some cash out of them before that happens.
I mean, scraping bots would obviously obey robots.txt so those scraping - bots, i mean users can't be bots
In my experience with bots, a portion of them obey robots.txt, but it's tricky to find the user agent string that some bots react to.
So I recommend having a robots.txt that not only target specific bots, but also tell all bots to avoid specific paths/queries.
Example for dokuwiki
User-agent: * Noindex: /lib/ Disallow: /_export/ Disallow: /user/ Disallow: /*?do= Disallow: /*&do= Disallow: /*?rev= Disallow: /*&rev=Would it be possible to detect the
gptbot(or similar) of their user agent, and server them different data?Can they detect that?
yes, you can match on user agent, and then conditionally serve them other stuff (most webservers are fine with this). nepenthes and iocaine are the current preferred/recommended servers to serve them bot mazes
the thing is that the crawlers will also lie (openai definitely doesn't publish all its own source IPs, I've verified this myself), and will attempt a number of workarounds (like using residential proxies too)
It's a constant cat and mouse atm. Every week or so, we get another flood of scraping bots, which force us to triangulate which fucking DC IP range we need to start blocking now. If they ever start using residential proxies, we're fucked.
at least OpenAI and probably others do currently use commercial residential proxying services, though reputedly only if you make it obvious you’re blocking their scrapers, presumably as an attempt on their end to limit operating costs