Lemmy.world status update 2023-07-05

server load is too low, everyone upvote more stuff so i can optimize more

edit: guess there is some more work to be done 😁

Test:

Upvote if you can see this comment. 👍

The change is noticeable. Good job guys.

Thanks for the updates.

This is why having a big popular instance isn't all bad. It helps detect and fix the scaling problems and inefficiencies for all the other 1000s of instances out there!

Wow. So much smoother today.

Great work.

You dropped this 👑

You guys had better quit it with all this amazing transparency or it's going to completely ruin every other service for me. Seriously though amazing work and amazing communication.

My upvote can go through fast now

Good work

I love the smell of updates in the morning.

Thank you guys for your awesome work!

Also to other people: DONATE TO FOSS PROJECTS. If 50.000 people donate only 0.5€, we have 25.000€ for funding the servers, coding, motivating/ people etc. Just don't take a cup of coffee for 1 day. We are already 2 millions in Lemmy instances. We can build a decentralized world together!!

Boy does it feel good to have those reports and understand the work you guys do. It's really inspiring! Thanks for your hard work, everything has been silk smooth! This instance is really great, Lemmy and its devs are really amazing and I feel at home in a nice, cozy community.

I'm not sure wtf you just said, but lemmy.world feels very smooth today, so thank you for your continued hard work!

Am I getting this correct: the whole lemmy.world instance run in one single container on one single host?

So that's why it was so smooth today... Great work!

Submitting PRs is literally the most effective response that helps everyone who uses Lemmy. Thanks to you all.

This is better optimization than most enterprise devs will see in their lifetimes.

upvoting posts is so much more stable now, we might actually see more bean posts as a result

Upvotes are still getting rejected. Replies hang so I cancel out and it turns out they did post.

That said, browsing is pretty snappy and smooth. I know the kinks will get worked out eventually. Thanks for the update.

Edit: This now appears resolved minutes later. All smooth on my end.

This is why I love open source. The fact that a community can directly debug the code that's it's being hosted on and directly contribute the improvements back is just wild. Thanks for all the hard work @ruud@lemmy.world and the rest of the lemmy.world team! The site already feels much more responsive.

Literally a night and day difference in performance and stability! Thank you all for the hard work. To other users like me, consider reducing or replacing one of your lesser used subscriptions and directing that money to Lemmy. It’s much better served here if you ask me.

The server is absofuckinglutely flying today! It feels smooth and bug free!!! You guys are legends.

As a data engineer, I'd be interested in hearing more about the SQL troubleshooting.

EDIT: It looks like !lemmyperformance@lemmy.ml is a good place to subscribe to for more technical info on some of these performance improvements.

Also the Lemmy GitHub of course contains more information on bugs/enhancements/etc.

Good to see a heavy production server taking on the scaling issues. Thank you! To discuss Lemmy performance issues, there is a dedicated community: !lemmyperformance@lemmy.ml

Appreciate that these updates use the yyyy-mm-dd format :D

It now feels pretty good to browse and it now makes the experience of using Lemmy much more enjoyable. Having to spam the vote buttons was really annoying.

It's so smooth now; the speed difference is insane! You all are doing excellent work!

Thanks to all involved across the board. Great work all around 👏👏

Even though i'm not from this instance, this is such a nice way of keeping the users posted about changes. I wish more companies (I know this is not a company) went straight to the point, instead of using vague terms like "improved stability, fixed few issues with an update" when things are changed. I hope all instance owners follow this trend.

Can we have an update on the status of Lemmy.world and how close ties we are going to have with Meta's threads? Threads is going to support ActivityPub, but time has shown that this is an attempt to try to kill this open platform and eventually replace it with theirs once they get everyone in their ecosystem. (Embrace, Extend...extinguish) Mastodon has said today that they don't mind sleeping with vipers when their demise / dissolution is in Meta's best interest.

Please tell me we are defederating from Meta....or let us know what to expect

EDIT: I originally stated that Mastodon told them to fuck off, but I got confused with Fosstodon (who did that). Mastodon doesn't mind being in bed with Meta

Huge props to everyone working on the project. It's awesome seeing everyone work together and resolving issues so quickly!

Thanks for this very nice report.

Thanks for the detailed update and all the hard work you guys are doing!

Gadzooks! These are huge fixes. Compliments to the team, you guys pulled off a small miracle today.

How great is it to be a part of history in the making -

This is Web 3 in its fomenting -

Headlines ~5yrs:

The ending of Web 2 was unceremonious and just ugly. u/spez and moron@musk watched as their social media networks signaled the end of Web 2 and slowly dissolved. Blu bird’s value disintegrated and Reddit’s hopes for IPO did likewise. Twitter and Reddit dissolved into odorous flatulence as centralization fell apart to the world’s benefit. Decentralized/federated social media such as Mastodon and Lemmy made their convoluted progress and led Web 3’s development and growth…

This is how history is made, it’s ugly and convoluted but comes out sweeet…

I don't understand anything other than you worked diligently to make things smoother. Thanks to everyone for their wonderful work!

Lemmy's devs and the .world admins have done in a month what Reddit hasn't done in it's whole existence: having a smooth and almost bug-free experience.

Jerboa feels so damn FRESH to use now!

The site is running so much better now, thanks to all. BTW: Love these updates!

I'm very curious: does single Lemmy instance have the ability to horizontally scale to multiple machines? You can only get so big of a machine. You did mention a second container, so that would suggest that the Lemmy software is able to do so, but I'm curious if I'm reading that right.

Shouldn't the correct HTTP status code for a removed API be 410? 404 indicates the domain wasn't found or doesn't exist, 410 indicates a resource being removed

Awesome work - things seem to be running much more smoothly today.

Do you have anything behind CDN by chance? Looking at the lemmy.world IPs, the server appears to be hosted in Europe and web traffic goes directly there? IPv4 apparently seems to be resolving to a Finland-based address, and IPv6 apparently seems to be resolving to a Germany-based address.

If you put the site behind a CDN, it should significantly reduce your bandwidth requirements and greatly drop the number of requests that need to hit the origin server. CDNs would also make content load faster for people in other parts of the world. I'm in New Zealand, for example, and I'm seeing 300-350 ms latency to lemmy.world currently. If static content such as images could be served via CDN, that would make for a much snappier browsing experience.

Is it safe to use 2FA yet?

Whilst I'm aware that too many users on one instance can be a bad thing for the wider Fediverse, I think it is a great thing at the moment in terms of how well people are banding together to fix the issues being encountered from such a surge in users.

The issues being found on lemmy.world results in better lemmy instances for everyone and improves the whole Fediverse of lemmy instances.

I'm very impressed with how well things are being debugged under pressure, well done to all those involved 👏

Hey I can upvote now!

That's so awesome! Look at that GRAPH!

I'd volunteer to be a technical troubleshooter - very familiar with docker/javascript/SQL, not super familiar with rust - but I'm sure yall also have an abundance of nerds to lend a hand.

I just love the transparancy you guys are coming forward with. It's absolutely awesome! Thank you for that and for all the work you put in. It means a lot to me that you folks are taking the time to keep us updated. Much love!

You guys are absolute legends, thanks for the update!

It blows my mind with the amount of traffic you guys must be getting that you are only running one container and not running in a k8s cluster with multiple pods (or similar container orchestration system)

Edit: misread that a second was coming up, but still crazy that this doesn’t take some multi node cluster with multiple pods. Fucking awesome

You guys are absolutely amazing. So many thanks to you @Ruud and the entire admin/troubleshooting team! Thank you.

smoooooooooth! Keep up the good work!

Is it weird that I’m always excited to read the update posts?

Wow it is smooth as butter now. Great job ruud and team!

Installed Jerboa again and it feels smoother than Reddit itself, great job!

Things have been super smooth lately, thanks for all the work!

Awesome work. Any way other devs can contribute?

@ruud crazy impressive

I took a SM break for a few days, and it's running noticeably better today...I think. (:

Thanks a bunch for floating us degenerates.

You know there's something about dealing with the lagginess in the past few days makes me appreciate the fast and responsive of the update. It nice to see the community grows and makes the experience at Lemmy feels authentic.

You guys did an awesome job.

Really great job, guys! I know from my experience in SRE that these types of debugs, monitoring and fixes can be much pain, so you have all my appreciation. I'm even determined to donate on Patreon if it's available

It felt like I’d jinx us all if I commented but THANK YOU! This has been a wonderful experience today. Absolutely loving it and knew you just needed some time to work out the kinks that happen with fast growth.

As a Performance Engineer myself, these are the kind of performance improvements I like to see. Those graphs look wonderful. Nice job to all.

thankyouthankyouthankyouthankyouthankyou it's working very well today...

I like that the post goes in detail and allows us tech nerds to get hard watching this stuff instead of the regular corpo jumbo change log that consists of:

1. we uhh fixed some stuff so yeah good?

I love reading these! Thanks for all the work

I hope to start on some small contributions sometime next week. Stability has been noticeably better the last few days and I imagine it’s only going to get better.

Love the transparency. Thanks to the entire team!

Love these updates! Love the transparency!

A lot of this stuff is pretty opaque to me, but I still read through it because I love how detailed they are in sharing what's going on under the hood and how it relates to problems users experience. Kudos to you guys!

Proving that orgs do need DBAs!

Thanks for the updates! Seeing the details of how you work through these early issues is valuable to those of us thinking of starting an instance.

Great job guys! It really feels more responsive today

Great job. Everything seems to be working smoothly for me now. The past several days have been a bit rough but now it's all working.

Great work !!

It works so well, that's very refreshing

Would HAProxy work better as a load balancer? For work we switched due to some issues with NGINX; so far, the service has been much more consistent with pretty much no downtime, even when restarting server hosts.

Sweet. Yeah it is definitely snappier today. Feels great.

Definitely noticeable!

Also, the amount of communication here is AWESOME! Thank you for keeping us updated.

Ty for the hard work <3

Things have been very noticably improving every day since the big update! I've had very, very few problems today browsing on Jerboa.

Compared to days prior, things are running much better today. Page load speed, reliable post/reply/upvoting. I dunno if it's just happenstance but whatever knobs, levers and keystrokes you're manipulating, keep going the things! Thanks so much for a home absent of corporate BS.

Awesome, thanks!

Thanks for the update and thanks for the amazing collaboration you’re fostering!

I feel like Lemmy.World is leading the way for other instances to follow, and that’s a good thing to do. Even if it’s painful for everyone involved.

It's definitely a lot better today. Great work.

Responsiveness is significantly better than it has been at any point since Friday's Reddit exodus.

Well done...

It is much smoother than it was previously. Thank you!

Man I thought I noticed something different. For the past week or so I've gotten nothing but network error and Java errors in Jerboa which are completely gone now. Posts load almost instantly too. Appreciate the effort guys, was going insane.

The responsiveness of the website has felt drastically better today.

Wow I applied these PRs on my server as well, running waaay lighter now. And it seems the federation misses have cleared up! Bravo Lemmy.world team!

It's really fun to see these updates, and get insight into what you're putting into this instance. It's real obvious that it's a labor of love.

Seems a lot faster today - great work!

Now that's a satisfying couple of charts!

This was cool to read.

Everytime I open a post and go back to previous page it scrolls back to top. Is this fixable? Im on windows 11, chrome.

Amazing work, thank you so much!

I agree that everything is a lot more stable!

Was just noticing how much smoother it is this morning. Great work!

Nothing feels more exciting than constant update, communication and the potential of lemmy!

I'll be honest I don't know what any of this means but what I can say is I absolutely love the transparency of all of this. It's so refreshing and maybe I'll start learning more about what I'm looking at because I'll keep seeing it. Great work!

Ever since I joined Lemmy I had problems posting replies but that kink seems to have been ironed out. Thanks!

I'll be honest, I think I understood three words of your OP. Sounds like the general gist though is Woo so great 😃

Thanks for all your efforts, glad someone knows what they're doing

My god, it's fast right now. Don't touch anything.

Good job.

Thank you guys, you are doing an amazing job. These are great times for the internet and you are a huge part of it!

Well done guys!

Awesome, no more errors on my end! Thanks for the hard work, donated as a small token of thanks for the team here.

This update is a difference like night and day. Very impressive!

good vibes!! thank you for your work

Thank you so much for the hard work, time and money you spend into making lemmy.world run very smoothly. This much transparency is awesome for something that's being used so massively.

The instance seems to be much better. Posting and commenting is not taking as long and loading times are way better. I hope things can stay this good or even get better.

Thank you so much! I will be donating a few cappuccinos your way when my next check arrives. I really appreciate how awesome of a community you’ve brought together & all of the transparency with the updates (and the frequency) is astounding! Keep up the great work but don’t forget to take breaks :)

It’s very much helping the third party apps as well. Memmy is running way smoother now

That's a very phallic bandwidth image Dr Freud

Thank you so much for all the efforts! So happy to be part of this community.

Good job on troubleshooting!

Have you looked into possibly migrating to kubernetes or some other form of docker container management/orchestration system to help with automatic scaling and load balancing?

Thank you to everyone that helped.

Thank you for all your great work, and I really appreciate and enjoy the updates about the gritty details!

Thanks for your efforts and improvements. Keep it going ♥️🙏

Was waiting for this update. Woke up a couple hours ago and Lemmy was working great so I knew y’all put in some time. Thanks for your efforts!

Thank you for all the hard work and transparency as always! Everything is running perfectly knocks on wood

That's some pretty heroic shit right there.

You just took lemmy from something I'm willing to live with in the short term in hope it gets better, to something I am fully satisfied with.

Now let's grow so we can fuck it up all over again!!!

Awesome news. Thanks for all the hard work.

Thanks for all the work you've put into this!

It's great to see such transparent and informative updates on this.

Everything is feeling great so far. The only bug I'm encountering is that when opening a thread (in Firefox on desktop) it auto-scrolls down past the content to the replies.

The new updates made a ton of improvements. Thank you very much. It is near ideal now.

Please recommend people to update their app in a topic title. Connect couldn't even load a topic without failing out today. An update fixed it, but I had to manually force it because it didn't apply automatically.

This will drive people away. Literally none of the communities I subscribe to on world even seemed to have a new comment.

LEGEND!

Running so much smoother! You guys rock!

Amezing improvement! Lemmy felt so much more responsive suddenly today, now it makes sense why :3

Everything is much faster and smoother today. Thanks devs!

Things feel a ton better today. Thank you for all the hard work!

Seemed to be running a lot smoother, thank you Ruud!

Can confirm, having way less problem browsing lemmy right now. Thank you admins!

Wow! Your commitment and diligence is admirable!

You all are champions today. When I loaded this up today, I was very happy and impressed. I knew tweaks were made.

Thanks for the update! Things seem way speedier now ^^

Minor thing but over night both wefwef and Memmy clients are showing the wrong comment score (karma) against my profile, and given they are showing the same amount I assume it’s related to API fed data. Value was correct yesterday. Easy for me to confirm given I have only 2 dozen posts and the value has dropped to single digits.

Not a biggie, but figured I’d report it in case there was some issue causing that. Might be some optimisation around indexing or something has intentionally or unintentionally impacted that.

Otherwise the service feels much more stable currently. No timeouts today where it’s been very frequent the past few days. Nice job. 👍

Fantastic job in troubleshooting and submitting PRs! Improvements are very noticeable already :)

am i amazed by you all guys work?!

i absolutely am

Awesome update, great to see and feel the progress from all the hard work!

Upvotes appear to be working but I still can't post images. The post displays the infinite spinning loading circle.

Great work! Awesome to see how fast the technical side of the Lemmyverse is evolving and improving!

Thank you guys for all the hard work!!

Memmy is great! Been using it since about 2 weeks now. Really feel at home.

Things are looking a lot more stable, thank you (all) for the amazing work!

The changes are very much noticeable, thanks as always everyone!

damn bro, y’all coming in clutch to improve stability of this lemmy instance.

Good shit bros. Hope to contribute upstream and find more performance related bugs. I browsed the code for lemmy, and could not find any performance tests.

I can probably start there.

Woo hoo, nice graphs! Monitoring data that shows big improvements is always fun to see.

Amazing work @ruud@lemmy.world and admin team! Good to see some really awesome progress!

Cheers!

Really appreciate all the hard work going on behind the scenes! Feels night and day different after the changes. Also appreciate the transparency. Nice to see in this day and age.

Thanks for all your hard work. The change is absolutely noticeable.