You're viewing a single thread.
157
comments
Web manager here. Don't do this unless you wanna accidentally send google crawlers into the same fate and have your site delisted.
Wouldn't Google's crawlers respect robots.txt though? Is it naive to assume that anything would?
It's naive to assume that google crawlers respect robot.txt.
It'd be more naive to have a robot.txt file on your webserver and be surprised when webcrawlers don't stay away. 😂
157
comments