How did a CrowdStrike config file crash millions of Windows computers? We take a closer look at the code

The Register ^ | 23 July 2024 | Thomas Claburn

[ShadowAce]

[Gene Eric]

>> tux

exactly

[Ann Archy]

CROWDSTRIKE is a DEMOCRAT COMPANY! I sure hope only Democrats USE it!!

[kawhill]

Excellent thread. Spent over forty years of my life playing software engineer.

[xoxox]

error handling is a thing.

[Navy Patriot]

Thanks for posting this, I just searched my C drive for: C:\Windows\System32\drivers\CrowdStrike\ and nothing came up, so I assume that I have no Crowd Strike software on my machine, not that I should have.

However, now I know, Thanks again.

[T.B. Yoits]

error handling is a thing.

So is testing updates but apparently not to the team at ClownStrike.

[null and void]

Never be the first to download an OS, or any update, or new whiz bang software.

[pas]

It was not a mundane detail Michael.

[Alas Babylon!]

Keep in mind not every Windows system used Crowdstrike Falcon. In fact, it really was/is an enterprise-level tool.

There are many to chose from including:

• Cynet
• ESET Endpoint Security
• Trend Micro Apex One
• Symantec Endpoint Detection and Response
• Stormshield Endpoint Security
• CrowdStrike Falcon Insight
• Cybereason Total Enterprise Protection
• Malwarebytes Endpoint Protection
• Panda Endpoint Protection
• FireEye Endpoint Security
• Comodo Advanced Endpoint Protection

The one used the most is Symantec.

Falcon Pro is about $99 per endpoint and Falcon Enterprise is $190 per endpoint (both are annual subscriptions)

[Lockbox]

Just a happy dance because you use Linux….. 😂😂😂😂

[dhs12345]

This guy, an old MS NT developer has a couple of pretty good videos...

Essentially, CS were allowed to access and write code that operated the kernel. Oh, and the EU (European Union), didn’t allow MS to implement ways to protect the kernel... like was Apple were allowed. The EU were concerned about the monopoly that MS had over their OS.

https://www.youtube.com/@DavesGarage/videos

[dhs12345]

A cyber security guy recommended against using these types of software packages — Norton, MCaffee,.. He was more concerned about access to and security of and privacy than BSOD. But his point was well taken by me.

He said that the Windows version of protection is adequate — safety vs privacy.

[mmichaels1970]

So they came out with a workaround which required booting into safe mode and then deleting a sys file. Now, I was on vacation at the time and my company or personal stuff wasn’t affected, but here were the problems I immediately thought of in the workaround. I use hotel front desk clerk as an example because I was affected on the drive home when my hotel couldn’t make door key cards and had escort us and unlock our hotel room door with a master key.

Some of you way smarter folks can perhaps tweak my understanding of this wherever you’ve seen I’m going wrong:

1. You’re not going to be able to “remote into” a failing computer since it’s in a BSOD/boot loop. Gonna have to fix on site. You’re either going to have to:

a. Travel to the site and fix computers one by one.
b. Overnight and ship a new computer with the fix applied.
c. Talk a user through the workaround via phone.

2. Hard enough for ME to remember how to boot into safe mode let alone some front desk clerk at a hotel or manager at a bank. So good luck getting a non IT employee to boot into safe mode for you.

3. I believe once you boot into safe mode you’re going to need a local admin password for that machine. How many remote IT departments are going to let THAT one out over a phone call with a front desk clerk at a hotel. Most will guard that admin password with their lives.

So, assuming your IT support is remote rather than inhouse, you’re going to be dispatching a bunch of techs or shipping a bunch of systems all over the place for a while.

Am I getting this generally correct?

[dadfly]

yeah. a domino server crash doesn’t entirely explain why some of these companies have been down for days. but hey if you hire an IT company named ‘crowd strike,’ and let it install stuff across your enterprise without vetting, as an engineer, i don’t have much sympathy for you.

[dynachrome]

Putin and Xi laugh and take notes.

[mmichaels1970]

but hey if you hire an IT company named ‘crowd strike,’ and let it install stuff across your enterprise without vetting, as an engineer, i don’t have much sympathy for you.

I don't have a 100% grasp on all of the ins and outs, but I believe it's a bit more convoluted than that. As the end-user client company, usually you hire an IT support company (rather than staffing up your own inhouse IT department). That IT support provider assumes responsibility for protecting your network from cyber threats. If you ever get hacked, or some goofball clicks on an emailed hyperlink to let loose a bunch of russkie bits and bytes on your machine, you go after your IT support company and grill them for not adequately protecting you.

THAT IT support company decides to go with Crowdstrike which is one of several security software systems out there. Some dude at Crowdstrike messes up, IT support company has your systems set to auto update security software quickly (since these updates are usually responding to emerging threats), client company's computers all go poof.

Crowdstrike immediately says "oops, we messed up. But here's a little workaround that can fix the issue in minutes." Unfortunately, very few people actually sitting at these computers have the expertise OR security access to actually perform this workaround.

President of client company calls IT support company and threatens to fire them all if the issue isn't resolved. IT support company gets overwhelmed as they have more than one client doing this. Three tech guys quit cause they decide it's not longer worth the aggravation. Too much coffee ends up being drank...IT anarchy reigns.

Right now I blame Crowdsrike...and ONLY Crowdstrike.

[DouglasKC]

Wasn’t crowdstrike a huge part of the Clinton email scandal?

[bigbob]

You’d think Microsoft would know something about rolling out software updates...err..wait...