What happend to lemmy.ml? EDIT: Works with ipv6 disabled

Lemmy.ml did a server migration which was posted about 8 hours ago which was pinned on lemmy.ml meta. It appears in the comments that some users have experienced a 502 error, which was related to “iCloud’s Private Relay”. They were able to bypass the issue using a VPN, although I haven't had any issues accessing lemmy.ml on my pc. I also read something about AWS us-east-1 being down, so it could also be related to that.

us-east-1 must not be completely down, or I'd be having a much worse day!
- Saw it on https://lemmy.world/post/105778
@Sleeping I still wanted to follow a community there from my instance and I could not.
@soeren
- yeah our instance is having problems too with lemmy.ml communities
- Following communities could be another issue entirely, sometimes you have to subscribe then unsubscribe multiple times in order to get it to work.

It's weird being part of broad communities like technology or games again. On reddit I unsubbed all the default subs and subscribed to more niche things like 3D printing or weird games. It's quite a change to see people talking about things like mainstream AAA games. Intellectually I knew that a lot of people had to get hyped about the latest Bethesda game or Apple products or whatever, but I think part of me stopped believing it, because it still surprises me whenever I see it.

This appears to have somehow ended up in the wrong spot. I hadn't even clicked on this thread before, and yet here's my reply to a different one. Ah well.
- I've seen a couple of people mention this bug. I'll check if there's an issue in their bugtracker for it.
Retro games and patient games were pretty good subs.
- Yeah, I'm a mod of retrogaming and a few few other subs in that network. Best group of users ever. 320k people in just the one sub and I think as a team we had to take down maybe 3-4 posts a day? Maybe? I banned 2-3 people in a month, that's absolutely unheard of in a sub that size. I think the worst argument I saw the whole time was a running "the playstation one was not called the PSX" "Yes it was" "NO IT WASN'T!" "HERE IS AN ARTICLE, YES IT WAS." thing that we had to specifically make a rule about. Odd little drama, but other than that? No fights at ALL. I'm talking a report maybe every few days for something being off topic if we didn't get to it quick enough.
  The niche subs about fringe interests, even if they're popular ones, often bring out the very best people. And the patient/retro gamers are so very chill.
I also can't stand all these fuckin' normies. 🙃
- I know, right? Next thing you know I'll end up touching grass or something.

In short, yes it will take longer to "catch up".

The longer answer, be patient. This is an issue that will be resolved. I'm sure that the admins are doing their best to get the instance up & running ASAP.

From my experience during the great Twitter migration to Mastodon, you saw the same thing. Especially the "main instance(s)", mastodon.online and mastodon.social in this case, were overloaded to the brim. After a day or two, the sysadmins and developers were able to stabilize the instance.

The sysadmins in this case were responsible to keep the servers up and running (and scale up where needed). The developers because their code hasn't been "battle tested" for this level of interaction. So changes were needed, to optimize and squash bugs.

I hope this helps 😊

Edit: fat fingers, ie fixed typo

This is a fantastic point. I watched several migration waves from Twitter to the Mastodon/*oma/*key network over the course of Nov-Feb and each became less impactful as instances and developers understood better how that traffic would flow. Consider also that those platforms had also been around for a few years already.
Lemmy and kbin are very, very young still, so it will be great to see them develop over the course of the coming months. I expect the next couple weeks leading to the June 30 3PA closures on Reddit will be spent preparing for another wave.
- Oh definitely that the next couple of weeks will be spent preparing for a new wave. I wouldn't be surprised if that was the reason why Lemmy.ml was down. Yes I know that it was because of a server migration but, that server migration also didn't happen for no reason ;-)
  The first migration wave to Mastodon was very bad. I still remember, that following a person was nearly impossible. Not to mention the fact that, even if you could follow someone, it didn't automatically meant their messages ended up in your feed.
  I know for example that version 0.17.4 of Lemmy (that was released 2 days ago) already has some database optimizations.

It looks like the fix for this is to disable IPv6 - not all of us can do that (or want to do that).

Lemmy.ml will need to fix their AAAA record likely.

Not sure where they host but AWS is currently in the middle of a pretty big outage so it could be related.

After doing a WHOIS, it appears that Lemmy.ml is hosted on OVHCloud.
Edit:
As follow up on that, OVH cloud does appear to be doing some maintenance on their infrastructure: https://www.status-ovhcloud.com/

Used Reddit for 13 years, tried out Kbin and Lemmy yesterday and settled on Lemmy.

Long story short, I'm going back to Reddit.

There needs to be ONE site, Lemmy.com, that people goto. This entire thing about having .whateveryouwant is VERY off putting. Most internet users have been trained to be extremely wary of odd or unusual things, so having anything besides .com/.net/.org will turn away a huge portion of users.

I initially setup an account on Lemmy.world, then realized that I couldn't migrate it to another server and that when I deleted that account on that server all my comments were deleted.

Deciphering the distributed nature of it took me, a relatively tech-friendly person, almost the entire day and several 'What the fuck?' posts. I now understand it more. There are some very low-level guides that have been haphazardly put together, but there absolutely needs to be a MUCH smoother guide/explanation to this whole thing. That learning process will turn people away for sure.
BECAUSE I understand it more now, I'm left feeling VERY uncomfortable about my data security. If this is going to become a mainstream thing, as it reaches and before it gets to that critical mass of users, there's going to be SO. MANY. SECURITY ISSUES. There's no 2fa at all, hacking and user-account hacking is just going to run rampant, and I'm left wondering 'Where is my username and password actually stored?'. The answer, sadly, is wherever the dude who's running the instance/server is. In the 'Fediverse' your server instance might be hosted in a US or EU data center with proper digital and physical security, or it could be Joe Blows basement in Iowa running off a NAS. The easy-to-see future here is that Lemmy will fail to attract a critical mass of people because they'll initially arrive, after a few months their instances will just cease to exist/get shut down/the hosts will decide its no longer a fun hobby to do.

With a large corporation, they have the staff and resources to secure and maintain the servers physically and digitally, and keep staff up-to-date on current infosec threats and get out in front of them. Beyond that, if there IS a breach, they have the ability to recognize it, understand the legalities and requirements of reporting it, and can be held accountable by regulatory bodies. Joe doesn't have the resources to really maintain and keep a server running, nor the knowledge of his responsibilities for keeping the data safe digitally or physically.

On top of that, if Joe's basement loses power/gets hacked/Joe decides he's moving to San Fransisco and can't bring his NAS with him and the server goes down, and that's where my instance is hosted well there goes my entire account/comments/data.

Finding and subbing to communities is painfully difficult. It should be one-click, but somewhere I need to goto an external list, find what I want, and then copy/paste the URL into the search... and then 50% of the time, it doesn't work. This is an understandable growing pain and can likely be fixed by UI/UX upgrades, but for now it's a definite turn-off.
There simply is no content. I'm not a creator, I want content aggregated for me, and I've gotten used to having a single place to get it from that floods me with thousands of different articles/memes/posts/etc every minute. Until the user base arrives in one single place and starts generating content, there's no reason for most people like me to be there as by far the larger number of users never create anything at all and only exist to consume the content generated.

Thats not the point of being federated. It is very simple to understand once you get over the learning curve. The analogies to countries/states communicating to each other but being seperate helped me
- See, this is yet another issue with Lemmy. This is not the post I replied to. I replied to https://lemmy.ca/post/636721, "Redditors, how do you like Lemmy?", and yet it posted it in a completely different topic!
Sites don't store passwords, they store password hashes. There is no reason to give any personal info you aren't comfortable giving. You can use the site just fine without posting any
- Hacking an account is still a valid concern though for various reasons , and hashes can still be used against password lists. Additionally, Two factor authentication is a necessity for sure. Now don’t get me wrong, I completely understand this feature is coming and that this is a developing service but many of these concerns do seem valid to me.
There needs to be ONE site, Lemmy.com, that people goto. This entire thing about having .whateveryouwant is VERY off putting.
The main feature of a federated system is that there's no one "owner" of the entire system; each admin uses their own servers and their own domain names. There's nothing stopping a server admin from using a .com domain name...
There’s no 2fa at all
There's basic TOTP 2FA now: https://github.com/LemmyNet/lemmy/issues/2363 but it hasn't been released yet (it's only available in the latest server beta). "Next-gen" TOTP (Webauthn/FIDO) is coming later: https://github.com/LemmyNet/lemmy/issues/3059
‘Where is my username and password actually stored?’. The answer, sadly, is wherever the dude who’s running the instance/server is. In the ‘Fediverse’ your server instance might be hosted in a US or EU data center with proper digital and physical security, or it could be Joe Blows basement in Iowa running off a NAS.
You should get to know your server admin, then. You have the freedom to pick any server you like :)
There simply is no content
Have you subscribed to many communities yet? You can browse other Lemmy servers to find interesting communities, and subscribe to them on your instance.
- On a similar note, would it be possible to have something like "sharding" where one server has multiple synched copies on several people's machines? So lets say if one machine goes down for whatever reason there are others to still serve content? This could also help with distributing load across multiple machines so its less stressful on one, and we don't have situations such as whats going on with lemmy.ml now, where so many users are joining it that its frequently down and subscription statuses are stuck on pending
A large open source project is generally more secure than a closed source one as it has more developers working on it and more security experts reading the code or testing it. If you want an example look at how many developers are working on Linux vs Windows and there security history. Once lemmy is big enough it has the potential to be more secure than things like Facebook or Twitter even.
You apparently don't understand how password hashes work. Nobody is storing your password on any well implemented website. You can't just grab someone's password if there is a breach, provided the people who built the website aren't completely incompetent.
With these misunderstandings and the fact federation was confusing to you I am not sure I believe your claim that you are a "tech-friendly" person. You sound like someone who has above average exposure to technology rather than someone who is actually skilled in it. There is nothing wrong with this in itself but please don't claim to be proficient when you aren't.
I’m not sure how you’re being downvoted, you make some great criticisms of the fediverse. Maybe staying small is better for the communities than reaching a critical mass as that only attracts corporate sponsorship (VC/ads/etc.). Working out the security should be the initial goals for any interested programmers who venture over.

I get 502 in germany with mobile data and wifi

Are you on an Apple product, since that could be related to the “iCloud’s Private Relay” issue, since you're getting a 502 error?
- i tried with pixel 7 and macbook but both didnt work. Private Relay is disabled.

It's down for me too - southwestern U.S.

AWS Outage should be resolved now (it impacts my job, I would know).

IMO, looks like something else is up with this.

Apparently disabling ipv6 should make it appear again.

Working for me (West US). It is actually working really well. I think this is the fastest it has ever been.

Edit. Interesting I tried from another location and I am getting a 502 Bad Gateway error. I tried to do a ping and Lemmy.ml is resolving to a IPV6 address at that location. vs my current location is IPV4 address.

I disabled IPV6 and now it is working on the other location.

Not sure if it is an issue on my end or something on Lemmy.ml.

I just also tried this from the EU (NL to be exact). The moment I disable IPv6 it works, if I re-enable it, my OS prefers to pick IPv6 over IPv4 and thus I get a 502 error again.

Works AOK from here in NL

Curiously, I can't see lemmy.ml content in my Jerboa feed, but it's working in the browser.

Working for me (from UK).

EDIT: seems they migrated to a new server earlier today to handle their increased load. Maybe a hiccup related to that, or your DNS server is still pointing to the old instance or something (in which case it should eventually fix itself)

I haven’t yet looked at the Lemmy architecture of the server, but it should be fairly straightforward to have a redundant region on AWS and use Route53 for the failover.

I noticed that there's one thread on lemmy.ca, made by an account on lemmy.ml, that I just absolutely cannot reply to. It hangs, and nothing gets posted. I'm able to reply to other threads in the same community.

How exactly are these things structured? Like, could that issue be related to lemmy.ml doing its migration, even if the topic was posted on another instance? It was very weird behavior.