1y ago

Delta CEO says CrowdStrike-Microsoft outage cost the airline $500 million

www.cnbc.com

Delta Air Lines CEO Ed Bastian said the massive IT outage earlier this month that stranded thousands of customers will cost it $500 million.
The airline canceled more than 4,000 flights in the wake of the outage, which was caused by a botched CrowdStrike software update and took thousands of Microsoft systems around the world offline.
Bastian, speaking from Paris, told CNBC’s “Squawk Box” on Wednesday that the carrier would seek damages from the disruptions, adding, “We have no choice.”

92 comments

- Which didn't work.
  
  It worked but there was a $10 convenience fee.
- Technically, it was a $10 gift card for each IT technician, so that could have been a whole $100!
  Not so bad after all
- No, only the partners did
Bastian said the figure includes not just lost revenue but “the tens of millions of dollars per day in compensation and hotels” over a period of five days. The amount is roughly in line with analysts’ estimates. Delta didn’t disclose how many customers were affected or how many canceled their flights.
It's important to note that the DOT recently clarified a rule that reinforced that if an airline cancels a flight, they have to compensate the customer. So that's the real reason why Delta had to spend so much, they couldn't ignore their customers and had to pay out for their inconvenience.
https://www.kxan.com/news/can-you-get-compensation-if-your-flight-was-delayed-or-canceled-by-the-crowdstrike-outage/
So think about how much worse it might have been for fliers if a more industry-friendly Transportation Secretary were in charge. The airlines might not have had to pay out nearly as much to stranded customers, and we'd be hearing about how stranded fliers got nothing at all.
Why do news outlets keep calling it a Microsoft outage? It's only a crowdstrike issue right? Microsoft doesn't have anything to do with it?
- It's sort of 90% of one and 10% of the other. Mostly the issue is a crowdstrike problem, but Microsoft really should have it so their their operating system doesn't continuously boot loop if a driver is failing. It should be able to detect that and shut down the affected driver. Of course equally the driver shouldn't be crashing just because it doesn't understand some code it's being fed.
  Also there is an argument to be made that Microsoft should have pushed back more at allowing crowdstrike to effectively bypass their kernel testing policies. Since obviously that negates the whole point of the tests.
  Of course both these issues also exist in Linux so it's not as if this is a Microsoft unique problem.
  
  There's a good 20% of blame belonging to the penny pinchers choosing to allow third-party security updates without testing environments because the corporation is too cheap for proper infrastructure and disaster recovery architecture.
  Like, imagine if there was a new airbag technology that promised to reduce car crashes. And so everyone stopped wearing seatbelts. And then those airbags caused every car on the road to crash at the same time.
  Obviously, the airbags that caused all the crashes are the primary cause. And the car manufacturers that allowed airbags to crash their cars bear some responsibility. But then we should also remind everyone that seatbelts are important and we should all be wearing them. The people who did wear their seatbelts were probably fine.
  Just because everyone is tightening IT budgets and buying licenses to panacea security services doesn't make it smart business.
  
  The crowdstrike driver has the boot_critical flag set, which prevents exactly what you describe from happening
- The answer is simple: they have no idea what they are talking about. And that is true for almost every topic they are reporting about.
  
  But... the BSOD!
- It was a Crowdstrike-triggered issue that only affected Microsoft Windows machines. Crowdstrike on Linux didn't have issues and Windows without Crowdstrike didn't have issues. It's appropriate to refer to it as a Microsoft-Crowdstrike outage.
  
  Funny enough, crowdstrike on Linux had a very similar issue a few months back.
  
  I guess microsoft-crowdstrike is fair, since the OS doesn't have any kind of protection against a shitty antivirus destroying it.
  I keep seeing articles that just say "Microsoft outage", even on major outlets like CNN.
- I don’t know that Microsoft has OS goodwill. People use it because the apps are there, not because Windows has a good user experience.
  
  The driver is wqhl approved, but the update file was full of nulls and broke it.
  Microsoft developed an api that would allow anti malware software to avoid being in ring 0, but the EU deemed it to be anti competitive and prohibited then from releasing it.
  
  I think what I was hearing is that the CrowdStrike driver is WHQL approved, but the theory is that it's just a shell to execute code from the updates it downloads, thus effectively bypassing the WHQL approval process.
- Because Microsoft could have prevented it by introducing proper APIs in the kernel like Linux did when crowdstrike did the same on their Linux solution?
- Its sort of like calling the terrorist attack on 911 the day the towers fell.
  Although in my opinion, microsoft does have some blame here, but not for the individual outage, more for windows just being a shit system and for tricking people into relying on it.
Pretty sure their software’s legal agreement, and the corresponding enterprise legal agreement, already cover this.
The update was the first domino, but the real issue was the disarray of Delta’s IT Operations and their inability to adequately recover in a timely fashion. Sounds like a customer skimping on their lifecycle and capacity planning so that Ed can get just a bit bigger bonus for meeting his budget numbers.
- Negligence can make contracts a little less permanent.
  
  Delta was the only airline to suffer a long outage. That’s why I say Crowdstrike is the kickoff, but the poor, drawn-out response and time to resolve it is totally on Delta.
- Couldn't agree more.
  And now that this occurred, and cost $500m, perhaps finally some enterprise companies may actually resource IT departments better and allow them to do their work. But who am I kidding, that's never going to happen if it hits bonuses and dividends :(
  
  We just lost 500 million - we can't afford that right now! /s
  
  According to The headhunters are constantly trying to recruit me for inappropriate jobs it is starting to get traction with companies and they are starting to actually hire fully skilled it departments. Opposed to the ones merely willing to work for near minimum wage which is what they had before.
  In some ways it won't really make a difference because fully staffed up I.T departments also needs to be listened to by management, and that doesn't happen often in corporate environments, but still they'll pay the big bucks so that's good enough for me.
  
  Fucking lol.
  
  According to The headhunters are constantly trying to recruit me for inappropriate jobs it is starting to get traction with companies and they are starting to actually hire fully skilled it departments. Opposed to the ones merely willing to work for near minimum wage which is what they had before.
  In some ways it won't really make a difference because fully staffed up I.T departments also needs to be listened to by management, and that doesn't happen often in corporate environments, but still they'll pay the big bucks so that's good enough for me.
- I wasn't affected by this at all and only followed it on the news and through memes, but I thought this was something that needed hands-on-keyboard to fix, which I could see not being the fault of IT because they stopped planning for issues that couldn't be handled remotely.
  Was there some kind of automated way to fix all the machines remotely? Is there a way Delta could have gotten things working faster? I'm genuinely curious because this is one of those Windows things that I'm too Macintosh to understand.
  
  All the servers and infrastructure should have "lights out management". I can turn on a server, reconfigure the bios and install windows from scratch on the other side of the world.
  Potentially all the workstations / end point devices would need to be repaired though.
  The initial day or two I'll happily blame on crowdstrike. After that, it's on their IT department for not having good DR plans.
  
  There was no easy automated way if the systems were encrypted, which any sane organization mandates. So yes, did require hands-on-keyboard. But all the other airlines were up and running much faster, and they all had to perform the same fix.
  Basically, in macOS terms, the OS fails to boot, so every system just goes to recovery only, and you need to manually enter the recovery lock and encryption password on every system to delete a file out of /System (which isn’t allowed in macOS because it’s read only but just go with it) before it will boot back into macOS. Hope you had those recorded/managed/backed up somewhere otherwise it’s a complete system reinstall…
  So yeah, not fun for anyone involved.
- Maybe not.
  https://youtu.be/byZHIoqi8oo
Don't worry everyone... Each and everyone of the CEOs involved in this debacle will earn millions this year and next and will eventually retire with more money they could possible spend in 10 lifetimes
If anything, they'll continue to fall upwards completely deserving even more money
- Additionally, don't worry, they'll just shift more costs onto the consumer and ultimately widen their profit-margins in no time.
  Perhaps Boeing can save the airline industry a little more by lowering the costs of their planes by removing another bolt and jerry-rigging flight software onto an antiquated platform.
Yeah... Maybe don't put all your IT eggs in one basket next time.
Delta is the one that chose to use Crowdstrike on so many critical systems therefore the fault still lies with Delta.
Every big company thinks that when they outsource a solution or buy software they're getting out of some responsibility. They're not. When that 3rd party causes a critical failure the proverbial finger still points at the company that chose to use the 3rd party.
The shareholders of Delta should hold this guy responsible for this failure. They shouldn't let him get away with blaming Crowdstrike.
- So you think Delta should’ve had a different antivirus/EDR running on every computer?
  
  I think what @riskable@programming.dev was saying is you shouldn't have multiple mission critical systems all using the same 3rd party services. Have a mix of at least two, so if one 3rd party service goes down not everything goes down with it
  
  Adding another reply since I went on a bit of a rant in my other one... You're actually missing the point I was trying to make: No matter what solution you choose it's still your fault for choosing it. There are a zillion mitigations and "back up plans" that can be used when you feel like you have no choice but to use a dangerous 3rd party tool (e.g. one that installs kernel modules). Delta obviously didn't do any of that due diligence.
  
  If I were in charge I wouldn't put anything critical on Windows. Not only because it's total garbage from a security standpoint but it's also garbage from a stability standpoint. It's always had these sorts of problems and it always will because Microsoft absolutely refuses to break backwards compatibility and that's precisely what they'd have to do in order to move forward into the realm of, "modern OS". Things like NTFS and the way file locking works would need to go. Everything being executable by default would need to end and so, so much more low-level stuff that would break like everything.
  Aside about stability: You just cannot keep Windows up and running for long before you have to reboot due to the way file locking works (nearly all updates can't apply until the process owning them "lets go", as it were and that process usually involves kernel stuff... due to security hacks they've added on since WinNT 3.5 LOL). You can't make it immutable. You can't lock it down in any effective way without disabling your ability to monitor it properly (e.g. with EDR tools). It just wasn't made for that... It's a desktop operating system. Meant for ONE user using it at a time (and one main application/service, really). Trying to turn it into a server that runs many processes simultaneously under different security contexts is just not what it was meant to do. The only reason why that kinda sort of works is because of hacks upon hacks upon hacks and very careful engineering around a seemingly endless array of stupid limitations that are a core part of the OS.
  
  Alternatively, they could have taken Crowdstrike’s offer of layered rollouts, but Delta declined this and wanted all updates immediately to all devices.
Good. They’ve been stealing from their customers for decades; this is fuckin’ karmic.
- Also, maybe don't put all your eggs into one single basket, from an infrastructure perspective.
Sure, but they did send a $10 Uber Eats gift card, so you gotta take that into account.
- 499,999,990. Could be worse.
  Edit: oh I see someone else beat me to it
- Make it per affected device and then we can talk.
Good thing they got that $10 Uber Eats card.
Aw that’s a shame. Poor rich company.
Crowdstrike offers layered rollouts, but some executive declined this because they want the most up to date software at all times.
- Not for the rapid update that broke everything.
  See post incident report:
  How Do We Prevent This From Happening Again?
  Software Resiliency and Testing
  Improve Rapid Response Content testing by using testing types such as:
  Local developer testing
  Content update and rollback testing
  Stress testing, fuzzing and fault injection
  Stability testing
  Content interface testing
  Add additional validation checks to the Content Validator for Rapid Response Content.
  A new check is in process to guard against this type of problematic content from being deployed in the future.
  Enhance existing error handling in the Content Interpreter.
  
  Rapid Response Content Deployment
  Implement a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base, starting with a canary deployment.
  Improve monitoring for both sensor and system performance, collecting feedback during Rapid Response Content deployment to guide a phased rollout.
  Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed.
  Provide content update details via release notes, which customers can subscribe to.
  Source: https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
womp.
womp.
Lol
Revenge for the times they messed up my travel plans.
But at least a door didn't just fall off a plane. That's already fixed in windows 11 the shitwagon release.

92 comments

How Do We Prevent This From Happening Again?