Skip Navigation
Advice on how to deal with AI bots/scrapers?
  • In my experience, git forges are especially hit hard

    Is that why my Forgejo instance has been hit twice like crazy before...
    Why can't we have nice things. Thank you!

    EDIT: Hopefully Photon doesn't get in their sights as well. Though after using the official lemmy webui for a while, I do really like it a lot.

  • Advice on how to deal with AI bots/scrapers?
  • I did find this github link as the first search result, looks interesting, thanks for letting me know the term "tar pit".

  • Advice on how to deal with AI bots/scrapers?
  • Not gonna lie, the $3900/mo at the top of the /pricing page is pretty wild.
    Searched "crowdsec docker" and they have docs and all that. Thank you very much, I've heard of crowdsec before, but never paid much attention, absolutely will check this out!

  • Advice on how to deal with AI bots/scrapers?

    Long story short, my VPS, which I'm forwarding my servers through Tailscale to, got hammered by thousands of requests per minute from Anthropic's Claude AI. All of which being from different AWS IPs.

    The VPS has a 1TB monthly cap, but it's still kinda shitty to have huge spikes like the 13GB in just a couple of minutes today.

    How do you deal with something like this? I'm only really running a caddy reverse proxy on the VPS which forwards my home server's services through Tailscale. "

    I'd really like to avoid solutions like Cloudflare, since they f over CGNAT users very frequently and all that. Don't think a WAF would help with this at all(?), but rate limiting on the reverse proxy might work.

    (VPS has fail2ban and I'm using /etc/hosts.deny for manual blocking. There's a WIP website on my root domain with robots.txt that should be denying AWS bots as well...)

    I'm still learning and would really appreciate any suggestions.

    24
    "When I first tried this hearty potato stew, it was at my uncle's birthday party..."
  • Luke Smith's based.cooking aims to solve exactly that.
    Might be missing a bunch of things but it's always worth at least checking out.

  • br*tish rule
  • I'm something of a fartist myself

  • the bouny

    !

    no idea who the artist is 😞

    0
    sailorzoop zoey @lemmy.librebun.com

    Hey, I'm a 3D artist, bass player, and a bad programmer. I'm self-hosting everything I can, and have been using Linux since Ubuntu 14.04. (Arch btw exclusively since 2019) My first account on here was over at .ml about a year before the whole Reddit shtstorm. (@princesszelda, yoink) Also on blahaj @snugglebutt@lemmy.blahaj.zone

    Posts 2
    Comments 7
    Moderates