TIL in 1997, a Reply All storm took down all of Microsoft's internal Exchange system
TIL in 1997, a Reply All storm took down all of Microsoft's internal Exchange system
It started off with an employee sending an email to a distribution list called "Bedlam DL3" asking to be taken off the list. With 13,000 recipients and everyone replying all with, "Me too!" and other messages, it was estimated that over 15 million messages were sent through the system in an hour. This crashed the MTA service due to a recipient limit. Each time the MTA service recovered, it would attempt to resend the message again which lead to a crash loop.
As a result of the incident, the Exchange team introduced message recipient limits and distribution list restrictions to Exchange, which is something we all use today!
More on the story here: https://techcommunity.microsoft.com/t5/exchange-team-blog/me-too/ba-p/610643
cross-posted from: https://techy.news/post/2224
I love this story. We have preventive measures in place because of chaos like this. Back in the early 2010s I was doing a VA internship at my college's Veterans Affairs office. We would help other veteran students manage their GI Bill and other VA benefits as well as communicate events for our student vet community. The school used a student veteran distribution list just like this story but for a much smaller group of users.
One day the college's VA office sent out an email that a prominent anti war vet was coming to speak at the college which sparked a lot of controversy apparently with a good portion of the members of that DL. So much so that a reply all chain began and my student email inbox was blowing up from all the outrage. If I remember correctly, the event was ultimately canceled because so many student veterans were complaining.
Why would anyone, but especially a vet, not be anti-war? A vet has seen war up close. Why would they want more of it?
Not all vets have seen combat, even if they have deployed. But aside from that most of these vets were early 20s that did one enlistment and are fresh out of active, so their life experience up until that point was high school to military to college. Not all of them, but a significant amount. That being said, even at the time I would've found the absurd response funny, except it meant more work for me that week to draft communication on behalf of the office.
might be some vets feel anti-war = anti-vet (it doesn't)
Like the two people configuring an auto reply when they went on holidays ...
You can figure out the rest I guess, fun even if it wasn't 15 million mails in an hour :-)
A coworker of mine and I had auto replies set up back in the day to do this on purpose. If either of us emailed the other we'd take down the server. It went on for months and we passed each other notes instead of emailing. It was our own private cold war.
He accidentally sent a company wide email instead of his contact list that excluded me. We were banned from auto replying to each other after that.
A while ago someone at my company resigned and then sent a leaving message to all 2000 employees.
The reply all storm that followed was hilarious. I have no idea how general staff was able to use the all staff distribution group.
This is fascinating! Great TIL!
Back when I was in university, I worked IT support there on the side. One day, a teacher wanted to send a mail to one project group, but accidentally send it to the whole university. Every student, every employee. We didn't reach 13k people but it was a few thousand.
The thing is, in Outlook (which was used for school mail) the default reply button which is looking simply like arrow, was the reply-to-all one. Reply to sender was hidden a few clicks away. Needless to say, this caused similar issues. With the first people just politely trying to tell said teacher he might have maken a mistake, then people went in replying asking people to stop using reply-to-all, and it didn't take long for hell to break loose after that.
To make matters worse, a few smartasses ran some scripts putting the whole receiver list on all kinds of spam advertisement lists, causing a flood of spam send to everyone simultanously with all the reply-to-all-replies. And then people replied to those too. Guess they figured they wouldn't get caught with everyone receiving mails from everywhere. They did tho, and got seriously reprimanded.
The server automatically changed from instant delivery to synchronising every 5 min, but that still meant hundreds of mails every 5 min. Eventually we had to turn off the mail server to make it stop as trying to tell that many people to stop replying is impossible and it clearly wasn't going to die out on it's own.
It was a long day at work, and one I will likely never forget. But I feel like any bigger sized company that has excisted for some longer time has had their own version of this issue by now. I never understood why'd they make reply-to-all the default, instead of reply-to-sender with the to-all version as a smaller button next to it... At least they now added the warning in Outlook "your distribution group has X amount of people, are you sure" or something along those lines when sending to distribution groups of a few dozen or more...
Good old bedlam DL3