Skip Navigation

I made a little guide on blocking scrapers from your site that I hope will help

nova.mkultra.monster /tech/2025/01/14/don-t-just-ask-ai-to-stay-away-block-them.html

I was scouring the indie-web earlier, and found a pretty useful list of bots to add to your robots.txt. But, since I'm not convinced that this is enough to keep them away, I also figured out a simple way to at least potentially completely block them from accessing your websites.

6 comments
6 comments