AI companies are violating a basic social contract of the web and and ignoring robots.txt
AI companies are violating a basic social contract of the web and and ignoring robots.txt
As unscrupulous AI companies crawl for more and more data, the basic social contract of the web is falling apart.
![The rise and fall of robots.txt](https://lemmy.world/pictrs/image/d58d347c-eaf6-475a-bfe8-f71a8eb9140e.jpeg?format=webp&thumbnail=256)
You're viewing a single thread.
What social contract? When sites regularly have a
robots.txt
that says "only Google may crawl", and are effectively helping enforce a monolopy, that's not a social contract I'd ever agree to.36 0 ReplyI had a one-eared rabbit. He was a monolopy.
17 0 ReplySounds like a Pal name lol
1 0 ReplyOnly if its model is a Lopunny missing an ear
2 0 ReplyI was thinking of a short lil bunny wearing a top hat and monocle with one ear sticking out of the center of the top hat but that works too
1 0 Reply
When sites regularly have a robots.txt that says “only Google may crawl”
Is that actually true?
If so, why would they do that?
6 0 Reply