[Update: Failed again] Update to 0.18.1-rc.1 tried and rolled back
We've upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.
(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)
The main issue we saw is that users can't login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)
Also new account creation didn't work.
I'll create an issue for the devs and retry once it's fixed.
Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!
Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won't be on 0.18.1 any time soon I'm afraid..
@ruud@lemmy.world just wondering if you have considered setting up a second, beta, instance of lemmy.world open to the public?
With all the performance issues with 0.18.1, it's highlighted that there needs to be a way to stress test these updates before applying them to the main instance.
Suddenly my Jerboa won't change from local and I can't change the sort. Not sure if this is related, but it was working fine yesterday after I updated the app. But no longer this morning.
Damn, Im glad you tried, and thanks for the update! I just really want the front page fixed without changing to page 0 lol. That and the 3/6/12 hour filter option.
Keep us updated. Im bouncing back and forth between LW, KBIN, and .ml for now though
any recommendations on other instances I should try outside the lemmy-verse and kbin-verse?
appreciate the transparency. how are things looking in the back in lemmy.world (server wise)? will we get to a point where it wont require complete rollbacks on the state when a botched update gets rolled out?
Thanks for the transparency. Maybe it’s a good idea to have a test instance and some test cases/validation done there before updating the main instance. This is a regular process in any software/tech company/stack.
Testing should never be done directly on the prod instance.
I had a weird issue after it upgraded. My browser (Firefox on Android) showed the Lemmy UI but showed no posts. I checked my profile to see if I was logged in, and it showed I was logged in as someone else!
I cleared my cache and refreshed, it showed I wasn't logged in. After I logged in it was fine, and showed my expected posts and my actual profile. In Liftoff app my account worked fine the whole time, didn't need to log in again.
Great! I'm getting a bug, when I upvote a comment or post, the "loading animation" takes forever to complete. If I refresh the upvote is there. It reminds me to the bug we had with comments and new posts.
Jerboa is almost unusable for me at the moment. A lot of crashes, pages not loading, messages not sending etc. But I know that's inevitable at times. Restarting my phone seemed to make it worse. But I know you guys are on it! 👍
I'm so new, I'm a baby, I have no idea what I'm doing. I did just create my account, I tried using the app but it wouldn't let me. Saying server version too low.
I'm sorry for my ignorance, but is this what that means? And this means we can't use the app yet right?
Also, thank you for all of this. I will be here for it all, I'm staying forever! I already love this so much more than anything else 🥰
Running one of the largest and most vibrant Lemmy instances certainly makes this deployment a great test case for the devs and the system overall. @ruud@lemmy.world this community really appreciates you!
Kind of a related question. If we find bugs or anything, where can we report them? I have tried looking for a contact us page but couldn't find it anywhere.
I’ve found from hopping around some other instances that have upgraded to 0.18 that it is still pretty buggy. It does seem to be giving more information about the errors, instead of just failing like in 0.17, but spend any time browsing on those instances and you’re bound to be inundated with JSON and query errors. It also seems to get worse the longer you browse.
The UI changes are nice, and I do appreciate not having my feed auto-updating constantly, but I think you’d be making the right choice to hold off on upgrading until they can iron 0.18.1 out all the way. I’m not super knowledgeable about TS and Rust, but as a user it seems that switching from WebSocket created/shined a light on Lemmy’s issues with caching in general.
I managed to visit when the server was 0.18.1-rc. I didn’t see any posts and the selector for Subscribed, Local and All was “empty”, as in nothing was selected and I could not select anything.
After the update I can no longer log in or interact with Jerboa. Hopefully y'all resolve this soon, as I am really digging Jerboa, but thanks for letting us know what's going on, and keeping up the good work.
Thanks for the update! (I mean the communication regarding the update) 0.18 moved away web sockets right? Maybe that leads to different DB access patterns causing locks on such a big instance as lemmy.world.
I'll admit this is probably something I can Google, but is there any scale-out options for a given Lemmy instance, or is it designed to be hosted in a single server?
I would like to manage my expectations. Yesterday we were looking at 0.18.1 being sent out in about week from the devs. Was 0.18.1-rc.1 what we thought they'd have ready? Do we still think the 7-10 day window is likely?