So after we've extended the virtual cloud server twice, we're at the max for the current configuration. And with this crazy growth (almost 12k users!!) even now the server is more and more reaching capacity.
Therefore I decided to order a dedicated server. Same one as used for mastodon.world.
So the bad news... we will need some downtime. Hopefully, not too much.
I will prepare the new server, copy (rsync) stuff over, stop Lemmy, do last rsync and change the DNS. If all goes well it would take maybe 10 minutes downtime, 30 at most.
(With mastodon.world it took 20 minutes, mainly because of a typo :-) )
For those who would like to donate, to cover server costs, you can do so at our OpenCollective or Patreon
Thanks!
Update The server was migrated. It took around 4 minutes downtime. For those who asked, it now uses a dedicated server with a AMD EPYC 7502P 32 Cores "Rome" CPU and 128GB RAM. Should be enough for now.
I will be tuning the database a bit, so that should give some extra seconds of downtime, but just refresh and it's back. After that I'll investigate further to the cause of the slow posting. Thanks @veroxii@lemmy.world for assisting with that.
Like many others, I came from Reddit and was initially hesitant to try it out, but I love this place so much! It really feels like the "worse" parts of Reddit have been skimmed off, and that definitely shows with how nice people seem here! Thank you so much!
I'm not sure how its being done as far as the technical aspects but Ruud has done a great job as admin upgrading the servers to keep up and anticipating the flow of new users.
The same admin also has experience with a mastadon.world server that experienced lots of growth from Twitter users leaving over musk moves. So essentially we have a good admin as far as I can tell and it's not his first rodeo. Part of the reason I chose this server
For less tech-savvy newbies (like me), in case there is some confusion affecting your urge to engage/donate... My friend gave me a great explanation:
Lemmy the platform is planet Earth
“Instances” like lemmy.world, lemmy.ml, beehaw.org, etc. are like the different countries on Earth
When someone signs up, the user picks one instance to be a part of, like how an Earthling becomes a citizen of a country
If you register at lemmy.world, that means your home instance/ “home country” is lemmy.world, but you can “travel” to lemmy.ml, another instance / “country”, to check out and subscribe to their community
When you subscribe to a different instance that’s not your home instance, you can still participate in their content, and other people will be able to see which instance / “country” you’re from
Each instance can have its own version of the same “subreddit”, so you can have a c/Memes in your home instance that is different from a c/Memes in another instance. But you can subscribe to both separately
c/[community name] is the naming convention used here I think like r/[subreddit name] on Reddit. If talking about a community in a different instance, it's c/[community name]@[instance name] so like c/memes@lemmy.ml
Donations will help with the cost of running lemmy.world only and not lemmy.ml, beehaw.org, etc.
Someone please correct any of this if any of it is wrong, I’ll happily edit
I really appreciate what you're doing, but I'm worried how this instance will continue scaling. What happens when it gets to 1 million users? 10 million? We can scale vertically only somewhat, but horizontal scaling seems to be limited to "just join a new instance 4head" and that just...doesn't have a good experience.
Performance is looking awesome, lemmy.world is responding very fast to community subscription requests and search is also very fast. My experience when using other instances was that search didn't work at all, hindering community discovery.
Went ahead and subbed on patreon. Hope that lemmy survives the growing pains and can develop some of the community that reddit had!
Also if there are any fellow former apollo users would def recommend checking out Mlem, its in testflight right now but seems to be working towards the experience that apollo gave on reddit.
I'm not an engineer or a dev - but requiring a 32-core, $2000+ CPU to support 12k users doesn't seem like it would scale well. Is this normal, or does the fediverse require more computational resources than a simpler setup like reddit? How would a fediverse instance with 100k users be maintained?
Would be awesome if you create some group chat (e.g. Discord?) and add sysadmins/devops to it. Would be more than happy to assist, especially if you have questions or need opinions.
I've been working as sre/sysadmin/devops for the past ~5 years and ~9 years of (Arch) Linux user. More than 1K Arch Wiki edits over that period of time.
do you plan to publish any of your scaling data? Some others might consider helping by running large instances and your learnings would be incredibly helpful.
I’m just another reddit refugee but I wanted to say thank you for your time, effort and money. As I slowly come to terms with federation I see why some are so passionate about it.
So, I just want to make sure I understand this as I am a new user from reddit. Instances are server based and cost money. Instances are Lemmy.World, Beebaw, Lemmy.Film, etc etc. These are all seperate hosted instances. Correct?
And donations would help pay for the server, ie lemmy.world?
Cool, lemmy.world seems to be one of the busiest with lots of new users, we should be making it clear to new users that they can sign up with ANY lemmy instance and still access lemmy.world communities though. gotta keep the load spread out.
Hello, i still doesn't quite grasp about the concept of federation and about how fediverse works.
But does it means that one instance can only run from one server?
Say
lemmy.world running on Server A
lemmy.ml running on Server B
User can register on whichever they want and can see the post from server A and Server B
But when Server A reach maximum capacity, can Server A scale up or distribute the load to multiple instances?
How can we solve the issue of computing power when more and more users migrate to using this services
Thank you 😀
Sorry if its a dumb question, but the whole Federation concept is still new to me. I created multiple account to log in to beehaw, mastodon, lemmy.world, lemmy.ml at first because i dont know that with one user, i can see other communities from another instances
I don't understand why a dedicated server is a good idea, when the only true way to scale is to use like Kubernetes or Docker and ECS Containers with scale?
Your just gonna run into more problems, you cannot vertically scale forever.
I'm not too familiar with Lemmy's codebase, but I am a devops engineer. Is the software written in any way to support horizontal scaling? If so, I'd be happy to consult/help to get the instance onto an autoscaling platform eventually.
I just hope this doesn't cause ossies down the line. A lot of us are from reddit, and it wouldn't shock me if a lot of new users go back to reddit regardless of the outcome.
I'd hate for this situation to put unessecary financial strain on this service
My submissions still hang indefinitely, but when I reload they are there. I dunno if this is something connected to lemmy's code itself or the server but it's a bit broken
Lemmy.world is fantastic, thanks for your efforts. It fit perfectly with all the criteria I had when choosing where to host my account.
That being said, I wish Lemmy.ml, the "main" Lemmy instance, more often registered communities created here. At the moment, most people just search for communities there and many of our own don't show up because no user from that instance interacted with our new and growing communities just yet - not only does this create a fragmentation issue, but given the massive load spike, Lemmy.ml is actually running a bit slow whereas Lemmy.world is handling posts better, making interaction easier specially when migrating users from Reddit or other places. For instance, my GameBoy community is ready, with users, and I'm about to post some good content - but as far as someone from Lemmy.ml is concerned, no such community exists.
Anyone looking to host something big should check out bare metal hosting like Datapacket, Reliablesite, FDCServers, etc. Down side is total lack of handholding and other cloud features and the fact that you can't scale up without redeploying on a new box, but the upside is ridiculously cheap bandwidth. The bandwidth cost is by size of pipe, not gigabyte transferred, and pipes upwards of 10gbps are affordable.
OVH and Hetzner are also worth looking at but aren't quite as cheap bandwidth-wise.