In case anyone was wondering, yes we were down for ~2 hours or so. I apologize for the inconvenience.
We had a botched upgrade path from 0.17.4 -> 0.18.0. I spent some time debugging but eventually gave up and restored a snapshot (taken on Saturday Jun 24, 2023 @ 11:00 UTC).
We'll likely stick to 0.17.4 till I can figure out a safe path to upgrade to a bigger (and up-to-date) instance and carry over all the user data. Any help/advice welcome. Hopefully this doesn't occur again!
I suspect almost everyone here understands how this happens and feels more sympathy than anything. We've been there, and with higher stakes. Thanks for taking the time to administer this instance.
No problem at all, the transparency is appreciated and it's understandable. There will be learning curves and hiccups with any new and evolving tech being pushed into the limelight (Lemmy, in this case).
Sigh, I tried it one more time. Same problem. Downtime this time was probably < 5 mins though so hopefully not noticeable. I am wondering whether the database is in some bad shape (no idea why/how). I guess we wait and retry with 0.18.1.
It sounds like lemmy.world also had issues upgrading to 0.18.1 rc1. I believe they skipped 0.18.0 because the captcha feature was removed in that version, but readded for 0.18.1. They're solutions may be helpful, assuming it was similar issues.