{"id":44179,"date":"2024-07-22T14:16:02","date_gmt":"2024-07-22T18:16:02","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=44179"},"modified":"2024-09-17T09:55:17","modified_gmt":"2024-09-17T13:55:17","slug":"crowdstrike-update-causes-bsod","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2024\/07\/22\/crowdstrike-update-causes-bsod\/","title":{"rendered":"CrowdStrike Update Causes BSOD"},"content":{"rendered":"

Rory Tingle et al.<\/a>:<\/p>\n

The ‘most serious IT outage the world has ever seen’ sparked global chaos today - with planes and trains halted, the NHS<\/a> disrupted, shops closed, football teams unable to sell tickets and banks and TV channels knocked offline<\/a>.<\/p><\/blockquote>\n\n

See also: Reddit<\/a>, Hacker News<\/a>, and Slashdot<\/a>.<\/p>\n\n

Qasim Nauman<\/a> (Hacker News<\/a>):<\/p>\n

\n

Frontier Airlines briefly grounded all flights on Thursday amid a major outage in Microsoft networks, which also knocked out some computer systems at low-cost carriers Allegiant Air and Sun Country Airlines.<\/p>\n

Microsoft said on the status page for Azure, its flagship cloud computing platform, that the problem began at 5:56 p.m. and affected multiple systems for customers in the central United States.<\/p>\n<\/blockquote>\n\n

Andrew Cunningham<\/a> (Hacker News<\/a>):<\/p>\n

\n

Airlines, payment processors, 911 call centers, TV networks, and other businesses have been scrambling this morning after a buggy update to CrowdStrike's Falcon security software caused Windows-based systems to crash with a dreaded blue screen of death (BSOD) error message.<\/p>\n<\/blockquote>\n\n

Sergiu Gatlan<\/a> (Hacker News<\/a>):<\/p>\n

\n

The list of services impacted by the outage includes Microsoft Defender, Intune, Teams, PowerBI, Fabric, OneNote, OneDrive for Business, SharePoint Online, Windows 365, Viva Engage, Microsoft Purview, and the Microsoft 365 admin center.<\/p>\n<\/blockquote>\n\n

Edward Zitron<\/a>:<\/p>\n

What’s happened today with Crowdstrike is completely unprecedented (and I’ll get to why shortly), and on the scale of the much-feared Y2K bug that threatened to ground the entirety of the world’s computer-based infrastructure once the Year 2000 began.<\/p>

[…]<\/p>

The problem here is systemic — that there is a company that the majority of people affected by this outage had no idea existed until today that Microsoft trusted to the extent that they were able to push an update that broke the back of a huge chunk of the world’s digital infrastructure.<\/p><\/blockquote>\n\n

Jowi Morales<\/a>:<\/p>\n

\n

Southwest Airlines, the fourth largest airline in the US, is seemingly unaffected by the problematic CrowdStrike update that caused millions of computers to BSoD (Blue Screen of Death) because it used Windows 3.1.<\/p>\n<\/blockquote>\n\n

Tim Hardwick<\/a>:<\/p>\n

\n

The cause of the failure has been identified as an update to Crowdstrike Falcon antivirus software installed on Windows 10 PCs, but Mac and Linux machines running the same cybersecurity software have been spared.<\/p>\n<\/blockquote>\n\n

Simon Sharwood<\/a>:<\/p>\n

CrowdStrike’s now-infamous Falcon Sensor software, which last week led to widespread outages of Windows-powered computers, has also caused crashes of Linux machines.<\/p><\/blockquote>\n\n

Tom Warren<\/a>:<\/p>\n

\n

CrowdStrike says the issue has been identified and a fix has been deployed, but fixing these machines won’t be simple for IT admins. The root cause appears to be an update to the kernel-level driver that CrowdStrike uses to secure Windows machines. While CrowdStrike identified the issue<\/a> and reverted the faulty update after “widespread reports of BSODs on Windows hosts,” it doesn’t appear to help machines that have already been impacted.<\/p>\n<\/blockquote>\n\n

Rui Carmo<\/a>:<\/p>\n

\n

This is why I keep telling people that third-party kernel extensions should be banned from production servers, period.<\/p>\n

And shipping LIVE cloud updates direct to endpoints, unchecked, without any canaries?<\/p>\n

[…]<\/p>\n

But since most of the affected systems are in a boot loop that may well require physical (or IPMI) access to the machine.<\/p>\n<\/blockquote>\n\n

Howard Oakley<\/a>:<\/p>\n

\n

The macOS version of the Falcon sensor uses a kernel extension (kext) on Intel Macs prior to Big Sur, but because of the limitations of kexts on Apple silicon, it now uses an endpoint security System Extension instead.<\/p>\n<\/blockquote>\n\n

Stefan Esser<\/a>:<\/p>\n

People pointing to EndpointSecurity framework in MacOS as the solution for the Crowdstrike problem are missing the point. ES is a typical Apple solution and basically means:anyone who can bypass it has to have exactly one exploit (chain) that will allow them to bypass ALL vendors<\/p>

Sure yes running drivers in user land has less likelihood of taking down the whole system but it also means their functionality is severely limited by what API the vendor provided. Apple is simply gatekeeper in one more area of their devices.<\/p>

It would be sufficient for OS protection to mark drivers that crash as dirty and if this happens repeatedly boot without the driver and\/or optionally allow a rollback to a previously not crashing configuration<\/p><\/blockquote>\n\n

M.G. Siegler<\/a>:<\/p>\n

\n

The EC obviously felt they were helping out third-parties by requiring Microsoft to continue to grant the same level of kernel access that they have. And perhaps this was even a good thing for end-users as these companies could cover security bases that Microsoft wouldn't, for whatever reason – security in general, of course<\/a>, has not been<\/a> a Microsoft strong suit, of late<\/a>. But there are also often unintended consequences of such actions. In this case, a third-party service with a single code-push could take out millions of machines overnight and thus, cripple key infrastructure around the world.<\/p>\n<\/blockquote>\n\n

Ben Thompson<\/a>:<\/p>\n

\n

Fast forward nearly two decades, and while Symantec and McAfee are still around, there is a new wave of cloud-based security companies that dominate the space, including CrowdStrike; Windows is much more secure than it used to be, but after the disastrous 2000s, a wave of regulations were imposed on companies requiring them to adhere to a host of requirements that are best met by subscribing to an all-in-one solution that checks all of the relevant boxes, and CrowdStrike fits the bill. What is the same is kernel-level access, and that brings us to last week’s disaster.<\/p>\n<\/blockquote>\n\n

Tavis Ormandy<\/a>:<\/p>\n

This strange tweet got >25k retweets. The author sounds confident, and he uses lots of hex and jargon. There are red flags though… like what’s up with the DEI stuff, and who says “stack trace dump”? Let’s take a closer look…<\/p><\/blockquote>\n\n

Patrick Wardle<\/a> (tweet<\/a>, Hacker News<\/a>):<\/p>\n

I don’t do Windows but here are some (initial) details about why the CrowdStrike’s CSAgent.sys crashed.<\/p><\/blockquote>\n\n

Aleksey Shipilëv<\/a>:<\/p>\n

“Professional programmers” focusing on CrowdStrike disassembly\/language is a coping mechanism that protects them from realizing that there is a remotely updated 3rd party kernel module that is deployed on significant part of the world. That is why real postmortems are important.<\/p><\/blockquote>\n\n

Bryan Cantrill<\/a>:<\/p>\n

\nThe CrowdStrike BSOD fiasco is extraordinary in its scale and scope; on Monday’s Oxide and Friends, \n@ahl\n and I will be joined by security researcher and \n@LutaSecurity\n CEO \n@k8em0\n to help us sort through the many layers of this mess<\/p><\/blockquote>\n\n

See also: xkcd<\/a>.<\/p>\n\n

Previously:<\/p>\n