CrowdStrike has published the RCA (read here) for the CrowdStrike outage on July 19th as well as provided an Executive Summary on their Guidance page.

________________________________________________________________________________

8.1.2024 Update:

Hello,

Last night, CrowdStrike sent out a Tech Alert covering their planned actions for channel files. There are three topics of note:

1. Crowdstrike has currently paused deployment of channel files while they investigate and implement changes to their update process.

2. As a result, CrowdStrike is currently not pushing the channel files required to get a machine out of Reduced Functionality Mode (RFM), which is currently impacting some Windows machines. If you wish to receive the channel file update to restore a device to full functionality, please email secops@mcnc.org so we can contact CrowdStrike and request this update be pushed.

3. Starting August 7th, CrowdStrike will introduce options to adjust the rollout of Channel Files. Currently, the control granted is basic, and MCNC recommends going with the CrowdStrike recommended default rollout. CrowdStrike has implemented enhanced testing on its end to ensure another channel file error will not happen. If you have questions about this new mechanism, please email secops@mcnc.org with possible meeting times.

RFM Windows Devices

CrowdStrike is currently not pushing out the channel files required to restore Windows devices to full functionality after the recent Windows patches. Devices will show as RFM, meaning that protection is reduced as CrowdStrike is no longer monitoring more sensitive Windows components. There is also a slightly increased risk of Windows devices in RFM mode crashing. CrowdStrike will begin sending the necessary Channel files out starting August 7th; if you would like to receive this file sooner, please contact MCNC at secops@mcnc.org. MCNC strongly recommends requesting these channel files to ensure your machines are stable and protected.

Channel File Update Configuration

A bad channel file push caused the July 19th Windows BSoD crashes. To correct this action moving forward, CrowdStrike has implemented the following:

• Enhanced testing.

• Staggered Deployments across all CrowdStrike devices.

• Monitoring on CrowdStrike’s end to detect any potential issues and roll back a bad channel file.

• Control for customers – more on this below.

• Notifications – CrowdStrike already sends release notes for agent updates, and they will start sending similar ones for Rapid Response Content as well.

Here is a quick explanation of some terms:

• Agent – This is the actual Falcon Agent. MCNC runs a n-2 schedule by default.

• Channel Files – CrowdStrike uses these to adjust the way the agent behaves and interacts with the system.

• Sensor Operational Channel Files – These cover how the agent interacts with the operating system and are needed to ensure proper functionality and avoid RFM instances.

• Rapid Response Content Channel Files – These change the logic of the agent to ensure it is protecting the device against novel and ever-evolving threats.

CrowdStrike has implemented the ability to control the deployment of Sensor Operational Channel Files and Rapid Response Content Channel Files, but the current implementation is flat in nature and has some considerations. This is the first pass from CrowdStrike, and they have indicated that they will release more granular controls in the future.

Options and MCNC’s Recommendation

Crowdstrike has added multiple deployment options to control when channel files are pushed to assets once updates resume on August 7th.

• Early Access: Updates are applied immediately following successful internal testing at Crowdstrike

• General Availability (Recommended): Updates are applied in phases following successful deployment to Early Access customers

• Pause Updates (Strongly Discouraged): Updates are not applied, resulting in degradation of protection

MCNC strongly recommends that all clients leave their Channel File settings on General Availability. This ensures minimal administrative overhead and maximum security. As the policy can only be applied to all devices in your CID, there is no current ability to create a test/dev/QA group that will receive early channel files. While this seems to be “the old way” of receiving these updates, CrowdStrike’s enhanced testing and monitoring around channel files, along with staggered rollouts across their global clients, provides additional protections. This will require no action on your part as this change goes into effect on August 7th.

CrowdStrike DOES NOT recommend and currently does not support Early Access or Pause Updates. Early Access permits you to receive Channel Files immediately after CrowdStrike has finished testing. Pause Updates halts the selected Channel Files (Sensor Operational or Rapid Response) from being deployed to your devices, resulting in slowly degrading security.

Looking forward, MCNC recommends you review your environment and consider what a test/dev/qa environment looks like. During onboarding, Host Groups are discussed. A common example would be for the internal IT teams to run n-1 while the rest of the environment runs n-2. As CrowdStrike adds the ability to assign Channel File updates to groups, MCNC can further configure that IT group to receive Channel Files before the rest of the environment.

If you have any questions or concerns, please reach out the MCNC Security Operations team:

Email/Ticket : secops@mcnc.org

________________________________________________________________________________

7.26.2024 Update:

If you are still recovering from the BSoD issue caused by the bad CrowdStrike push early Friday morning, there are two potential vendor-provided automated remediations available for your consideration.

CrowdStrike Automated Remediation

CrowdStrike has introduced a new recovery mechanism, which is opt-in. This new mechanism will attempt to update the detection logic of the CrowdStrike sensor so that it quarantines the offending file and prevents the system crash from occurring.

• The system being recovered must be hardwired to the internet (not connected to WiFi) to have the best chance of recovery.

• The system may take up to 5 reboots for this work.

  • There is a chance that this process doesn’t work, and the manual recovery process of booting into safe mode and deleting the 291*.sys file is required.

  • Microsoft has a step-by-step guide for Windows 10 and 11, with screenshots, in the following KBs.

    • KB5042421 – Windows 10/11

    • KB5042426 – Windows Server

• Some system hardware has been seen to respond better to this process than others, potentially relating to the drivers or other mechanisms of the specific hardware related to boot timings.

• CrowdStrike is not aware of any negative side effects of this process and has seen it work on hundreds of thousands of endpoints.

• If you have Bitlocker, this solution does not require the key like the manual recovery effort does.

• You will need to submit a ticket to secops@mcnc.org requesting this be enabled for your environment.

MCNC encourages you to consider taking advantage of this solution. It may aid in the recovery of server and VM infrastructure. Additionally, if you have remote users, this may enable them to self-recover by plugging into their home internet and rebooting until the system recovers successfully.

Microsoft Automated Remediation via USB Drive

Microsoft has an automated USB drive recovery solution noted in KB5042429, which covers how to address the problem via Windows PE or Safe Mode.

• You will need a USB Drive with at least 8 GB of free space.

  • This USB Drive will be wiped, so please backup any data.

• If you have BitLocker, you may still need the recovery key.

• Once the tool is created, when the device is booted in Windows PE or Safe Mode, it will run a PowerShell script, which deletes the offending machine and reboots the device.

This merely automates the manual process, so it has the same efficacy.

________________________________________________________________________________

7.19.2024 Update:

This link provides a list of domains identified on July 19, 2024, that impersonate CrowdStrike’s brand.

https://www.crowdstrike.com/blog/falcon-sensor-issue-use-to-target-crowdstrike-customers/

Some domains in this list are not currently serving malicious content or could be intended to amplify negative sentiment.

However, these sites may support future social-engineering operations.

CrowdStrike Intelligence recommends that organizations ensure they are communicating with CrowdStrike representatives through official channels and they adhere to technical guidance the CrowdStrike support teams have provided.