Posted on 08/22/2024 6:12:49 AM PDT by ShadowAce
The cause of the CrowdStrike mess has been revealed by the company, together with the steps it has taken to ensure nothing similar can happen again.
The company is facing a deluge of lawsuits over the estimated $5B worth of financial losses incurred by its clients, but the small print in its contract may protect it …
A huge mistake by cybersecurity company CrowdStrike last month caused a global IT outage on a massive scale, with airlines, banks, health services, and more affected – including some 911 centers.
Airlines were forced to ground flights, broadcasters were taken off-air, retailers were unable to accept payments, hospitals couldn’t book appointments, and much more.
What wasn’t known until yesterday was the exact nature of the faulty update, and how it was issued globally without the problem being spotted in testing. CrowdStrike has now explained both.
In brief, the company wanted to make it easier to issue new threat updates to client PCs. To do this, it used a new approach which allowed dynamic configuration of the threat-detection. Protected PCs watched for updates based on a template comprising 21 pieces of data.
The crash occurred when CrowdStrike issued a template instance which contained only 20 pieces of data, one less than expected.
Therefore, the attempt to access the 21st value produced an out-of-bounds memory read beyond the end of the input data array and resulted in a system crash.
That then brings us to the second question: How was this not picked up in testing? In short, because the testing wasn’t done with real data.
The selection of data in the channel file was done manually and included a regex wildcard matching criterion in the 21st field for all Template Instances, meaning that execution of these tests during development and release builds did not expose the latent out-of-bounds read in the Content Interpreter when provided with 20 rather than 21 inputs.
CrowdStrike says it has made changes to ensure the instances match the templates, and has added runtime bounds to ensure that even if there’s a mismatch, it won’t cause a crash. Finally, it will in future do a staged rollout, so any problem which does make it through will only affect a limited number of PCs.
CrowdStrike is also facing a slew of lawsuits from large corporations, small businesses, and even its own shareholders. Wired reports:
On July 29, Delta informed CrowdStrike and Microsoft of its intent to sue over the $500 million it claims to have lost as a result of the outage. A class action lawsuit has been filed by law firm Labaton Keller Sucharow on behalf of CrowdStrike shareholders, claiming they were misled over the company’s software testing practices. Another law firm, Gibbs Law Group, has announced it is looking into bringing a class action on behalf of small businesses affected by the outage.
However, CrowdStrike terms and conditions impose strict limits on liability, and Jonathan Cardi – a law professor specializing in civil liability cases – says that negating this may prove challenging.
Those hoping to recover financial losses will need to find creative ways to frame their cases against CrowdStrike, which is insulated to a great extent by clauses typical of software contracts that limit its liability, Cardi says. Though it may seem intuitive that CrowdStrike be on the hook for its mistake, the company is likely to be “pretty well-guarded” by the fine print, he adds.
I believe CloudStrike/DNC will skate, but get plenty of companies to switch to other software.
It was a stupid mistake--one a first-year CS student would make.
ClousStruck is used by major corporations be cause the number two vendor is ... the Microsoft Defender platform.
That in itself wouldn’t be so bad but the M$ tech support is just a really bad Indian call center.
So, it’s off to work we go to battle the bad guys with one hand tied behind our back.
Thanks to ShadowAce for the ping!
Yes.
My company is trying to force a security product they bought for Windows onto the Linux side of the house. It's not working out very well, and it's actually lessening our security.
For context, I can give you physical access, AND THE ROOT PASSWORD, to our Linux boxes, and you still will not be able to do anything to them.
I'm of two minds about that. Yes, their f*ckup was egregious, and they did a lot of stuff really poorly that enabled a rookie error to take down so many computers.
My company's computers were affected, servers and laptops, and I personally had to fix a lot of them. I speak first-hand.
But OTOH, their product (Falcon) has been a tremendous boon to our security operations, and I'd hate to see it go under. There are competitors, but CrowdStrike has really served us well in terms of what they are supposed to provide.
So I'm hesitant to throw the baby out with the bathwater.
But on the OTHER other hand, our damage was relatively small. So I can be more sanguine than others.
P.S. I hate software license fine print.
Many people who were affected by this had no agreement with crowd strike.
ClownStrike’s name is mud. Their reputation in the crapper. Responible corporations will look elsewhere for internet security.
What would prevent a similar problem with everyone’s bank accounts????
Small print will not protect — the other party to the contract must AGREE and must UNDERSTAND.
That said, this is immaterial. For years, decades, software companies have been protected from liability every time they get to court. There is no professionalism or engineering liability in software. Never has been. It has always been “Buyer Beware”.
I’ve done dozens of software contracts and the companies have no liability, not even under a commercial license that emphasizes a “fit for duty” responsibility.
So “Our programmers screwed you up with our not-so-foolproof testing. But our lawyers screwed you up first - with our very good fool-proof contract.”
Not bad. Two screws for the price of one.
“has been revealed by the company...”
Well, I don’t trust anything CrowdStrike says so I’ll draw my own conclusions.
“The scale of the disaster was because most major corporations use CrowdStrike”
They have been very aggressive with their marketing. Eventually I had to just block their email spam because I will never switch my company to those hacks.
This sort of ‘stuff really hurts companies.
Several of our clients were running the SolarWinds Orion product. We bailed on that and other than Pingdom (a pretty good product that does not need credentials, etc.) are done with SolarWinds.
No clients were running CrowdStrike software - this problem makes running that software more unlikely.
“For context, I can give you physical access, AND THE ROOT PASSWORD, to our Linux boxes, and you still will not be able to do anything to them.”
How is that? Special hardware? (What architecture?) How do *you* maintain them?
What happens if *you* lose a crucial cryptographic key, such as an FS-related key (e.g. LUKS passphrase), the Secure Boot key (if non-standard), etc.? Can you get to the UEFI menu (or equivalent) via the HW Reset button to turn off Secure Boot?
P.S. With *complete* physical access (no time limit, no interference, the ability to boot from USB, etc.) the root password is usually quite unnecessary.
GRUB is password-protected, so you cannot edit it.
Root is prevented from ssh or logging into the console.
Disks are encrypted, so USB (if available--we run 90+% virtual servers) won't work.
UEFI is not involved in the configuration, so it doesn't matter if it's turned off or not.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.