We have to move to object storage to have efficient image storage. We are currently at 70% of our disk storage.
We are sorry for the short notice and we don't know how long it will take though we believe it will take multiple hours.
Small post mortem :
The migration required us to be down - we thought we could run without images
It took much longer than initially anticipated
It broke at close to 2:30AM - 3 hours into the migration and I couldn't fix it so the server has been down doing nothing for hours now
We're booting back up from the backup made before the migration was attempted, we'll try another strategy.
There might've been some data loss in images, we're looking into it. In the meanwhile, if your profile picture or banner is broken, feel free to re-upload them.
Update :
Hi Beeple!
We're trying this again.
This time, Beehaw should remain available though no pictures will be able to be uploaded. The error will likely be weird because Lemmy will think it is possible but we will block the upload from happening.
We'll take a snapshot of the pictures before the migration in 60 minutes at 21:00 UTC - it should take around one hour, do the migration and testing on our own before shipping on Beehaw.
Once we've resolved all these kinks, Beehaw will momentarily go down and then back up with the migration complete without losing any old pictures.
Didn't go as planned of course, restored from backups, pre migration attempt. Thank you for your patience while we try to get all these moving parts working well together. Sorry for the troubles.
I once caused an AWS outage that impacted 20% of their customers in their largest region. They called my manager to ask why we were performing around 10k writes per second to a bucket. It was fun times
Thanks for the update and hope you have less trouble in the future! Don't worry about the downtime I really appreciate that here it's serving a clear purpose unlike Twitter lol
I appreciate the late night efforts and the clear communication. For me, Beehaw is a positive place I can visit, but there are other things I can do also, and I have no need for many 9s of uptime here. (I'm trying to reduce any pressure you & others might feel - perhaps not communicated it well tho, hence this addition.)
Every technical bump in the road we hit now is one we won't hit/will know how to handle quickly in the future! Thank you for doing what you do for Beehaw!
Yeah, moving to object storage is best to do now. Arguably, we should've done it sooner since the longer we've waited, the more it was gonna catch up to us and cost us in time and money.
It could, and will. Hopefully they are taking advantage of CDNs for image delivery so they aren't paying high egress costs and can keep it in slow, cheap, storage.
I'm honestly surprised that Lemmy hasn't embraced distributed, community, hosting. Many existing niche communities (outside of Lemmy) operate with the ability for others to run their service to serve up images and media, or to act as workers (By running the worker application or container) for computationally expensive operations like compression or encoding. Even gamificating it in the case of e-hentai.
Hard drives at home are incredibly cheap compared to cloud storage costs (even including networking, server, redundancy...etc hardware costs), but come with reliability concerns, which is where a distributed community becomes critical.
I feel like Lemmy definitely needs to embrace distributed computing in some fashion. I have no interest in hosting my own instance, but I'm not against running a docker image that would offload some of the processing requirements large instances need. It would just need to be relatively straightforward for me to setup
I did an ADHD, and misread as you saying you were turning off pictures for good, but given how much I'm enjoying the Beehaw community and the hard work you guys to keep it online, I wasn't even that upset about that!
A short, well telegraphed, partial outage is nothing in comparison!
I am utterly ignoranant of technological mumbojumbo. I was just trying to add a pic to the Creative sub, but nothing uploaded. Is that why? Can I stop trying to make it work?
Yes. Be patient. Assume that when Fediverse stuff is not working exactly right, it's not you, it's probably the Fediverse. These are early days of self-organized effort, like thousands of people trying to lash rafts and boats together in the middle of the ocean. They're busy trying to make sure the whole thing doesn't sink - don't worry about the photos.
With kindness, I very much suggest against dismissing both the technology and your ability to understand it by calling it "mumbojumbo". Don't let the engineers make this stuff something only they can understand and work with.
They will ban your IP if you direct link to images and bypass their tracker links. If you use a VPN, be sure to clear your cache before changing your IP or the ban may carry over.