While the impact of the recent CrowdStrike-driven outage was widespread and still being felt throughout multiple industries, many have asked for assistance in better understanding what happened, what they can do to recover, and how they can prepare for and better defend against this happening to them in the future.
What is CrowdStrike Falcon?
CrowdStrike Falcon is a widely used extended detection and response (XDR) platform, that detects indications of compromise and anomalous activity across an organization’s IT estate – similar to the combination of Defender and Sentinel used by most of our clients. Products like this are used to detect suspicious behavior as it occurs, allowing for quick (often automated) response to stop (or reduce the impact) of incidents in progress and facilitate rapid follow-up.
What Happened?
CrowdStrike Falcon is a SaaS-based platform but relies on deploying agents to workstations and server endpoints as part of its protection methodology. Most deployed software is regularly updated, and the CrowdStrike agents are no exception. In this case an update to the “content update agent” was faulty and corrupted the Windows OS kernel (core operating system files), causing all systems that received the update to crash. This left the impacted systems in a state that requires recovery activity before being able to return to service.
How can these systems be brought back online?
The recovery procedure is straightforward and well-documented but can be labor-intensive at scale. Affected devices must be booted up in safe mode, and the corrupted file removed. Unfortunately, this requires direct access to systems (or the hypervisor layer the systems are running on).
How do we prevent this from happening again (or to us)?
System updates should always be staged, especially for business-critical applications. Deploying and testing updates to non-production systems before allowing them to apply to production is an important part of a robust resilience strategy. Organizations should know the risk impacts of losing various systems, and IT resilience should be prioritized accordingly.
How can Atmosera help?
Whether you are actively trying to recover systems, want to proactively address your organization’s resilience posture, or even move from CrowdStrike to Microsoft Defender, Atmosera can help.
As always, we’re here for our Managed Services customers around the clock, as well as available to engage on a project (Professional Services) basis, to help guide and execute system recovery plans, or even just provide skilled talent to help get your existing recovery efforts completed more quickly.
Our Security Professional Services team is also available to help you assess your resilience posture, and plan out, and execute changes to the way your organization manages its risk and potential areas of exposure.
If you’re ready to move to Defender and/or Sentinel, we are also ready to help you deploy these systems for your teams to manage, or to make them ready for our Managed Security team to run on your behalf.