2y ago

Lemmy self-hosters. What is your image cleanup process?

I'm self-hosting the docker containers and I noticed the pictrs directory is steadily growing because of the cached images. Does anyone know if it gets cleaned up automatically or are hosters running scripts to clean it up after a certain amount of time? The install guides make no mention of it from what I can find.

28 comments

IIRC, I've read comments elsewhere that pictrs caches for 6 months, but I can't independently verify. I hope this gets a broader answer because I'm still on the fence about getting an instance set up for myself and some small communities.
- I believe the activity table in Postgres is retained for 6 months (although I’m purging mine daily) and the pict-rs cache is 168 hours (1 week).
  
  I knew I read something was kept for 6 months ;)
  Glad to see that even here, the best way to get the right answer on the internet is to provide a wrong one.
  
  Only 1 week? That should be fine. Thanks!
  
  How do you purge daily? Also, does that delete any post history or anything in a similar vein?
- Related note, pictrs is super cool. Its like an OSS imgur backend, but no one really talks much about it or its potential.
- It would probably be worth it to have that period be configurable by instance admins...
  
  I think it’s configurable inside pict-rs’s configuration file. I haven’t messed with it though. I’m also not sure if pict-rs has an API that lemmy can use to configure that.
Honestly, If I can get posts to stay synced up, that will be a good day for me.....
Seriously, federation/sync issues, are not fun.
- Name checks out
  
  Hah!
- I've had lot of issues with lemmy.ml. I just unsubscribed from everything over there since zero comments were federating over to my instance.
  
  I noticed that they'll show up eventually where "eventually" could be like, 10-12 hours.
  I suspect that they're just absolutely slammed to the point they can't actually push the federated content out to subscribers because EVERYONE is subscribing.
  Might be an architectural thing due to not having a sufficiently scalable job queue/worker thread infrastructure, or just like, not enough CPU cycles to do it.
  
  Beehaw has been my bigger problem child.
  However, tonight it's smooth as butter. Things are syncing, I'm getting alerts.
  Could be due to some of the maintenance I did earlier too.
I'm just letting mine do whatever it wants, got plenty of local storage. If/when I have storage issues I'll add an s3 bucket, pretty easy to modify the entrypoint for pictrs to pass s3 connection info in the docker-compose deployment.
- S3 support is a good thing, thanks for mentioning it.
Remote images are not cached or proxied right now as far as I know. Edit: seems I was wrong and there is some image caching happening. For sure for the small image thumbnails, but also sometimes for other pictures, but it seems very inconsistent.
Your growning pictrs directory might be also due to the extremely verbose default logging that Pictrs (and the Lemmy backend too btw) uses.
- When I look in the directories, it's 100s of images that are definitely from posts. Maybe it only caches the images I clicked on?
  
  No, I was wrong and caching is happeing somehow, but not always. I think there might be a strict time-out or something like that for pict-rs trying to cache the images, which is why most images do not get cached in my experience.
I though instances only cached the text of submissions? I could see that ballooning to be insane pretty quick if the fediverse really takes off.

28 comments