Hello everyone,
unless you have been in an alpine cabinet with no internet connection, the news of Crowdstrike's outage has been hard to miss. A huge amount of businesses have been affected.
From airlines where several airlines canceled nearly all flights, to payment providers, banks, 911 service disruptions and even healthcare. Microsoft estimates that over 8.5 million devices were affected by the update - in my eyes a very low estimate. As a result the stock has been down 11% during Friday's trading session.
Dan Ives called it a black eye, and that it is importantly not a hack/cyber security threat. This notion has been thoroughly repeated by others on Twitter/X and some even compared it to the American Express Salad Oil scandal, where Buffett famously bought the cratering stock price and made a huge profit on the rebound.
However, as I am currently working in IT, this incident is a lot worse than if it would have been a security breach. Crowdstrike showed their incompetence and lack of checks in place. Microsoft breathed a sigh of release, because at least it wasn't them!
What happened
On Friday morning the company pushed an update, which caused Crowdstrike's Falcon Agent to crash Windows computers resulting in a blue screen of death and the PCs restarting over and over (bootloop).
The workaround was deleting specific sys files, which caused the Crowdstrike driver to crash. Tto be effective Crowdstrike is buried deep in the system, this caused a Windows Kernel panic and thus the computer wasn't able to restart.
The error was due to an invalid memory address (here are some technical details if you are interested: https://x.com/patrickwardle/status/1814343502886477857 )
This resulted for example in airlines shutting down, or reverting back to writing boarding passes by hand.
After the workaround was initially discovered it was posted in the knowledge base of Crowdstrike behind the login screen. An insane blunder given that many systems were down, so employees probably didn't have access to all credentials finding the fix on their phone. It was luckily re-posted by other media, but that was not the only blunder they made regarding their response here.
Their CEO George Kurtz posted on Twitter that customers remain fully protected despite the incident. That much is obvious, because the systems were offline. He then said that they deployed a fix.
The problem is that they just rolled back the update - and that the fix has to be made on the IT side. That can either happen with a PXE Server, where most PC's would be able to be quickly fixed, or if there isn't a PXE Server the IT team has to go into each system manually, delete the sys files and then update Crowdstrike once the PC has booted.
Their technical update from the company reads more like a liability and lawyer document than a real technical update.
https://www.crowdstrike.com/blog/falcon-update-for-windows-hosts-technical-details/
I know people who spent the whole night restoring hundreds of servers, having tens of thousands or even hundreds of thousands affected endpoints. With the whole IT in scrambles trying to restore the most essential business functions it will take a while until every one of them is working again.
Why this is worse than a breach/hack
A lot of Wall Street analysts said that this is only a hiccup and that because it was not a hack, that it isn't a big deal. I disagree. First the update bricked millions of PCs, requiring manual intervention to fix the problem. Secondly it shows a completely broken release process on the side of Crowdstrike, which is a lot less forgiving than a cyberattack - which is to be expected.
This is also not the first time the CEO was involved in a major update meltdown. In 2010 as the CTO of the antivirus software McAfee it caused a reboot loop of Windows XP PCs, also requiring manual repairs.
https://www.zdnet.com/article/defective-mcafee-update-causes-worldwide-meltdown-of-xp-pcs/
Five years ago, Crowdstrike released the 5.19 update resulting in blue screens as well and again requiring manual intervention.
A few months ago another Crowdstrike update broke Debian and Rocky Linux.
These incidents show a major problem in the internal release process of Crowdstrike. Just like with Solarwinds were malpractices went on for years, a single big incident can wipe out or significantly reduce the customer base.
The first instance where the update bug should have been caught, was on the developer side. It seems that there an update was pushed without a test. The second would have been an automated test in the CI/CD pipeline. The third instance should have been the Q&A team, that tests outgoing releases and the last instance should have been a staggered roll-out. Meaning that instead of pushing the update to every client, they would have pushed it to 1% of them and after about an hour pushed it to 10% etc.
This is especially important for a company like Crowdstrike, because you can not disable the automatic updates. The company has eroded all trust form a stability standpoint and that is hugely important in IT.
Crowdstrike is more in the premium offerings, and because it mostly worked - and there wasn't a huge incident before, IT managers were glad to roll them out. Well not anymore
Why this isn't a buy the dip moment.
Despite the 11% drop on Friday, the stock still trades at an incredible 22x P/S. Lofty expectations for a company, which will likely see major repercussions.
This also isn't similar to the American Express Salad Oil scandal. While putting a lot of pressure on the company it did not effect the core business of American Express. This breach does.
Given the premium nature of Crowdstrike new customers will rethink to either go with Crowdstrike or a different provider like SentinelOne. Crowdstrike will have to give huge discounts in order to convince new sign-ups, having to choose between obliterating their margins or their growth. Others will churn. An anti-virus software is less embedded into different systems than an authentication provider like Okta, so you can't compare the two breaches.
Switching costs are a lot lower. Okta already had to offer huge discounts, and it will be worse for Crowdstrike. Crowdstrike used to be like buying GE in the 90s - you couldn't get fired. Now not so much now. No IT manager will sign up with them unless they are a lot cheaper than other offerings, current contract negotiations will grind to a halt.
There will also be loads of lawsuits.
The damages are in the billions, and given that critical infrastructure was affected simple SLAs or TOS won't protect them, especially since it affected so many different countries.
Their insurance company also likely doesn't want to pay given that the nature of the outage wasn't a hack or breach but severe incompetence.
Furthermore this incident has gotten the attention of every government body - 911 services, hospital and airport were affected. Cybersecurity diversity will be discussed and Crowdstrike will likely need to pay damages and prove with costly audits and oversight teams that such a thing can't happen again.
Conclusion
Crowdstrike as the stock is priced to perfection. At 22x P/S it expects huge growth with nothing going wrong, but it just did. Either their growth or their margins will be absolutely obliterated and given their valuation it has a long way to drop. If lawsuits from different companies, countries and jurisdictions follow as well as new regulation, there is no way in my eyes that they can sustain such lofty valuations or growth expectations.
SentinelOne trades at half the P/S valuation and Palo Alto Networks at around 15x P/S. A small discount from SentinelOne might entice some companies to switch, I am taking a long/short approach on Monday.