Global Impact of the incident

Summary

On July 19th, 2024, Cybersecurity company CrowdStrike rolled out a sensor configuration update to Windows systems as a part of its regular operations. This update triggered a logical error and resulted in a system crash and blue screen on the Windows systems.  

From banking systems to airline operations, the infamous "Blue Screen of Death" plagued Windows computers worldwide. Corporate offices, banks, supermarkets and telecommunication services became inoperable, impacting daily operations and transactions. Airports in the US, Australia, Japan and India faced major disruptions with flights canceled due to this technology malfunction.  

In the wake of the widespread disruption, CrowdStrike issued a swift apology, acknowledging the seriousness of the situation and the impact it had on users across the world.  The company also assured ongoing efforts to identify the root cause of the faulty update.

The US Cybersecurity and Infrastructure Security Agency (CISA) released a alert warning about malicious actors attempting to take advantage of the vulnerable Windows systems.  

‍What exactly is this “Blue Screen of Death”?  

The Blue Screen of Death (BSOD) also known as a "Stop Error" is a critical system crash that occurs on Windows Operating systems. This dreaded sight occurs when a system encounters a fatal issue leading to abrupt restart and potential data loss to prevent further damage to the host machine.

Causes

Some of the causes for this error are as follows:

1. Computer drivers:  These are files created by the device manufacturers to enable the hardware to efficiently work with an operating system. According to Microsoft's blue screen error site, 70% of this Stop errors are caused by third-party driver codes.
2. Hardware and Software: Faulty RAM's, hard disk drive (HDD's), Solid-state Drive (SSD), motherboard, processors and incompatible software such as applications or programs may cause conflicts leading to a BSOD.
3. Overheating: If a computer overheats due to dust, malfunctioning fans, or overwhelmed hardware, it might display the BSOD.
4. Malware: A malware such as a PC virus that corrupts essential files and directories can also trigger a BSOD.  

‍Incident Analysis

According to CrowdStrike article, the technical incident analysis is as follows:  

CrowdStrike Falcon updates its configuration files known as "Channel Files" regularly to adapt to new threats. These files are part of the system's protection mechanisms. This process has been very normal since the beginning.  

In a Windows system, these Channel Files reside in the following directory:  

C:\Windows\System32\drivers\CrowdStrike\

These files have names starting with "C-" followed by a unique number (e.g., C-00000291-) and ending with ".sys". Importantly, while the .sys extension suggests kernel drivers, Channel Files are not actual kernel-level drivers.

Channel File 291, specifically designed to monitor how programs use named pipes (a standard Windows communication method) for suspicious activity, malfunctioned after an update. This update, aimed at identifying malicious named pipes used in cyberattacks, contained a programming error that caused Windows systems to crash.  

‍Affected Systems

Microsoft claims CrowdStrike's update impacted a substantial 8.5 million Windows devices which is less than 1 % of all the Windows machines and added that this could be the worst cyber event in history.  

According to CrowdStrike:  

• This issue impacts Windows 10 and later systems.
• Mac and Linux systems were unaffected.
• This issue is caused by CrowdStrike Falcon content update and not due to any malicious cyber activity.

Recovery and Remediation  

To expedite recovery for Windows devices affected by the recent CrowdStrike outage, Microsoft released an official tool specifically designed for IT admins. This tool addresses the BSOD (Blue Screen of Death) error caused by the faulty CrowdStrike update. While CrowdStrike offers a software fix, manual troubleshooting can be time-consuming. Microsoft's solution streamlines the process by creating a bootable USB drive for swift recovery of impacted machines.  

Microsoft's recovery tool streamlines the repair process for Windows machines impacted by the CrowdStrike update. To utilize the tool, IT admins will need to boot the affected system into the Preinstallation Environment (PE) using a bootable USB drive created by the tool itself. Once booted into PE, the tool automatically locates and removes the problematic CrowdStrike file, allowing the machine to boot normally. This method eliminates the need for local admin rights as the tool directly accesses the disk, bypassing the local Windows environment. However, for BitLocker-encrypted drives, the tool will prompt for the recovery key before proceeding with the repair.

CrowdStrike has released mitigation instructions for the systems that have been already impacted. These steps are as follows:  

• Boot Windows in Safe Mode or Windows Recovery Environment
• Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
• Find the file named "C-00000291*.sys" and delete it  
• Boot the host normally  

Here's a way a user can resolve this issue by utilizing the Blue Screen Troubleshooter available in the Get Help app. The process is as follows:  

• Open Get Help app on Windows.
• Type ‘Troubleshoot BSOD error’ (Blue Screen of Death) in the search bar of the Get Help app.
• Follow the step-by-step instructions provided in the Get Help app

Impact

According to ThreatMon, a claim on a dark web forum alleges a vulnerability in CrowdStrike software, potentially allowing unauthorized access to Microsoft 365 data on affected Windows machines. This information, reportedly including Microsoft account credentials, phone numbers, and personal details, is said to be offered for sale at $10,000.  

Capitalizing on the disruption caused by the CrowdStrike issue, the Handala Hack group reportedly launched a targeted phishing campaign against thousands of Israeli organizations. This campaign allegedly involved the use of the group's custom wiper malware and Fear, Uncertainty, and Doubt (FUD) tactics to compromise systems.

According to CrowdStrike, threat actors were observed distributing a malicious ZIP archive file targeting America-based CrowdStrike users.

‍Conclusion

Although, a swift response from both CrowdStrike and Microsoft rolled out a software fix and a recovery tool respectively, organizations impacted by the BSOD event may face a prolonged timeline to bring all affected systems back online. This incident serves as a stark reminder of our growing reliance on cyberspace, where internet infrastructure is no longer just a convenience but a vital element of modern society. Cybersecurity threats, like the recent global 'blue screen of death' event triggered by a faulty CrowdStrike update on Microsoft Windows systems, can have widespread disruptive effects at national and social levels. This incident also highlights the crucial role of robust cybersecurity measures and the importance of international cooperation in ensuring the resilience of our interconnected digital world.

Sources Cited:  

1. https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
2. https://www.crowdstrike.com/blog/falcon-update-for-windows-hosts-technical-details/
3. https://wat-not.com/tech/crowdstrike-updates-caused-global-outrage-affected-8-5-million-microsoft-device-users/
4. https://thehackernews.com/2024/07/faulty-crowdstrike-update-crashes.html  
5. https://status.cloud.google.com/incidents/DK3LfKowzJPpZq4Q9YqP
6. https://www.business-standard.com/industry/news/decoded-windows-10-crash-what-s-blue-screen-of-death-ways-to-resolve-124071900491_1.html
7. https://economictimes.indiatimes.com/magazines/panache/microsoft-outage-cause-explained-what-is-crowdstrike-and-why-users-are-getting-windows-blue-screen-of-death/articleshow/111858827.cms?from=mdr
8. https://www.malwarebytes.com/cybersecurity/computer/blue-screen-of-death  
9. https://learn.microsoft.com/en-us/troubleshoot/windows-client/performance/stop-error-or-blue-screen-error-troubleshooting  
10. https://www.globaltimes.cn/page/202407/1316373.shtml  
11. https://www.techradar.com/computing/internet/windows-blue-screen-of-death-crisis-what-we-know-so-far  
12. https://www.ndtv.com/world-news/windows-systems-restarting-throwing-blue-screen-of-death-due-to-crowdstrike-error-6138820
13. https://www.avg.com/en/signal/fix-windows-bsod
14. https://www.welivesecurity.com/en/cybersecurity/beyond-blue-screen-death-software-updates/  
15. https://www.livemint.com/technology/tech-news/microsoft-windows-outage-live-netizens-celebrate-international-bluescreen-day-blue-screen-of-death-crowdstrike-11721372942537.html  
16. https://www.hp.com/us-en/shop/tech-takes/what-is-blue-screen-of-death-windows-10  
17. https://timesofindia.indiatimes.com/technology/tech-news/microsoft-offers-fix-for-laptops-affected-by-crowdstrike-update/articleshow/111914628.cms
18. https://techcommunity.microsoft.com/t5/intune-customer-success/new-recovery-tool-to-help-with-crowdstrike-issue-impacting/ba-p/4196959  
19. https://www.news18.com/tech/windows-blue-screen-of-death-why-didnt-apple-devices-get-affected-by-crowdstrike-outage-8973631.html  
20. https://www.hindustantimes.com/world-news/microsoft-outage-which-industries-were-the-affected-in-global-it-chaos-101721438320867.html