On Friday, a security update from CrowdStrike caused a major IT disaster, leading to widespread system failures across various industries. Arjan Egges, an entrepreneur and university lecturer in computer science, discussed the incident in a video on his YouTube channel, ArjanCodes. Egges highlighted the fundamental issues that lie at the core of such failures and predicted that these problems will only intensify in the future.
Who is CrowdStrike?
CrowdStrike is a cybersecurity company that protects organizations from cyber threats such as malware and ransomware. Their platform, Falcon, utilizes AI and machine learning to detect, prevent, and respond to security threats in real time. Because Falcon operates at the kernel level of the operating system, it requires more permissions and fewer guardrails than typical software.
Recap of Friday’s Outage
Egges provided a recap of the Friday outage, explaining that a buggy update to the CrowdStrike driver caused Windows machines to enter a permanent boot loop. This resulted in significant disruptions, with airports halting flights, people unable to pay for groceries, hospitals unable to operate, and even coffee orders at Starbucks coming to a standstill.
The Industry’s Broken State
Egges didn’t hold back in his critique of the industry. “Our industry is broken,” he said. He questioned how a security product from CrowdStrike could cause such a massive failure and criticized Microsoft for allowing a product to impact the stability of their operating system so severely. He also called out businesses for viewing software development and maintenance purely as a cost, leading to short-term thinking and inadequate development processes.
Avoiding Such Disasters
Egges acknowledged that there are common practices that could help avoid such incidents. He referenced Dave Farley from Continuous Delivery, who mentioned four key practices: better testing to cover more edge cases, canary releases to a small group of users first, better observability to detect problems sooner, and the ability to roll back changes quickly.
The Fundamental Dichotomy
Despite these practices, Egges identified a fundamental problem: a dichotomy between building software with robust guardrails and the need for security software to act quickly with high permissions. Security software like Falcon needs extensive permissions to operate effectively, and in the face of imminent threats, there might not be time for thorough testing or staged releases. This creates a tension between the need for quality and the need for speed.
Things Will Get Worse
Egges warned that the situation will likely get worse. As IT systems become more embedded in every aspect of our lives, system failures will have increasingly devastating consequences. With more tools powered by AI and more code written by AI, our collective understanding of these systems diminishes, making it harder to prevent and address failures.
AI and Robotics Concerns
Egges highlighted the increasing involvement of AI in software development processes, from code reviews to cloud resource optimization and deployment. He also pointed out the growing presence of robotics in our lives, such as self-driving cars and robots in factories and restaurants. These advancements, while promising, also introduce new risks and potential points of failure.
Seeking Solutions
In concluding his video, Egges posed several critical questions: How can we ensure high-quality software? How can we implement rigid processes to manage single points of failure? How can we understand the systems that surround us to avoid becoming victims of their failures? Although he admitted he didn’t have the answers, he stressed the importance of addressing these issues.
Security Software Monoculture
People in the comments shared their thoughts: “I think the even more fundamental problem here is the security software mono-culture. I know CrowdStrike is big, but honestly, I was surprised when I heard in the news how broadly sweeping the impact was across companies and even across industries. If everyone’s using the same software, that provides a ripe attack vector for hackers.”
Another commenter said: “The Crowdstrike disaster hasn’t struck because they needed to move fast, but because they obviously haven’t tested this specific update on a single Windows machine. Because if they did, they’d immediately noticed it would crash. And they made a similar mistake already in April. That time it could be somewhat forgiven because it only occurred on two distributions of Linux which hadn’t been in their test matrix.”
Cybersecurity Challenges
Arjan Egges’ analysis of the CrowdStrike outage underscores the complexities and challenges of modern software development and cybersecurity. His call for better practices and a deeper understanding of the systems we rely on is a timely reminder of the potential risks that come with technological advancement. As we move towards an increasingly AI-driven future, ensuring the reliability and safety of our systems will be more crucial than ever.
Balancing Quality and Speed
What are your thoughts? How can software developers balance the need for thorough testing and quality assurance with the necessity of rapid response in security software? What are the potential risks and benefits of integrating AI more deeply into the software development process, and how can we mitigate the risks? How can we improve our collective understanding of complex software systems to better predict and prevent failures?
Check out the entire video for more information on ArjanCodes’ YouTube channel here.
