AI companies are violating a basic social contract of the web and and ignoring robots.txt
AI companies are violating a basic social contract of the web and and ignoring robots.txt
www.theverge.com The rise and fall of robots.txt
As unscrupulous AI companies crawl for more and more data, the basic social contract of the web is falling apart.
![The rise and fall of robots.txt](https://lemmy.world/pictrs/image/d58d347c-eaf6-475a-bfe8-f71a8eb9140e.jpeg?format=webp&thumbnail=256)
4
crossposts
You're viewing a single thread.
All comments
187
comments
This is a very interesting read. It is very rarely people on the internet agree to follow 1 thing without being forced
13 0 ReplyLoads of crawlers don't follow it, i'm not quite sure why AI companies not following it is anything special. Really it's just to stop Google indexing random internal pages that mess with your SEO.
It barely even works for all search providers.
17 0 ReplyThe Internet Archive does not make a useful villain and it doesn't have money, anyway. There's no reason to fight that battle and it's harder to win.
3 0 Reply
187
comments
Scroll to top