Crowdpatching call-to-action: How micropatching can make it happen
I am a penetration tester. As the CEO of ACROS Security, I've been hacking into our customers' networks for 16 years, and I have a dirty little secret to share with you: To break into an enterprise network today, you can still follow the exact same process as you did in 1999.
You need follow just four basic steps: 1. Find a public exploit for a vulnerability that is less than four months old; 2. tailor the exploit to work with your remote administration tool; 3. mutate the exploit until VirusTotal doesn't recognize it (mind you, there was no VirusTotal in 1999, so you had to install antivirus locally); and 4. phish users in the enterprise until you're in.
If this sounds as if there has been practically no improvement in enterprise security since 1999, that's because it's true. So why is that?
The only relevant measure of information security is how difficult (or costly) it is for attackers to gain access to their targets. This hasn't changed. But wait, you say, tons of new security products are showcased at trade shows each year that promise total security using machine learning, complete detection and prevention, and other security magic. But when wearing my attacker hat, I have been confronted with most of these leading-edge security technologies, and they haven't made my job any harder.
Why is that? Why have we in information security—this huge global engineering collective motivated by billions of venture capital dollars and growing enterprise security budgets—been unable to do anything to break the simple four-step process above? There is a way forward. It's called micropatching deployed as a hotfix. But everyone in the community will need to pitch in if it's to work. It needs to be crowdsourced.
Infosec needs to refocus
The reason those four old-school steps continue to work is that the vast majority of the infosec industry is focusing on the wrong problems. It does so because the wrong problems also happen to be the interesting, challenging, entertaining, sexy problems, with apparent solutions that sound clever, innovative, and disruptive, with a distinct taste of silver bullet. These are the problems for which solutions will get headlines, attention, and funding.
Contrast that with the age-old, super-boring, most unappealing problem of infosec on which nobody enjoys working: fixing vulnerabilities. It may be boring, but it is be the main reason behind today's successful breaches.
In one sense, this state of affairs is understandable. After all, what developer would rather spend time analyzing and fixing security bugs in code than build new products or features? Fixing bugs means you have nothing meaningful to show for your work at the end of the day. At best, it lands you where you already thought you were before someone pointed out the bug to you. Furthermore, security bugs are a bit different from functional bugs. You can't simply publish a workaround on your FAQ page so that users that have the particular problem can look it up.
No, you're racing against attackers who want to exploit a bug that will affect all of your users. That means you must drop everything else (including your weekend with family), fix the bug, test the fix, and deliver it to your users' systems ASAP if you want them to have a fighting chance. If this doesn't sound unappealing enough, you also don't get a single thank you, much less any remuneration for your sacrifice.
Now, suppose that an altruistic developer fixed a nasty vulnerability and his product actually had a mechanism that let him deliver an update to all users before attackers were able to create a working exploit. It's a win, right?
Not so fast. On home computers or in small businesses, users and admins might just click OK and apply the update. But in the enterprise, things are more complicated. No admin wants to be the person who single-handedly took down an automobile production line, creating millions of dollars in losses. Before the enterprise can apply an update, it must go through rigorous testing on dedicated testing machines, and then it's gradually rolled out to production systems throughout the network. This process can take weeks or months. (The average time to apply updates in the financial services industry is six months.) So even when the diligent developer spends his weekend in the office to quickly deliver a fix, attackers still have months before users apply it to exploit the bug. This time frame between when a security fix is public available and when enterprises actually apply the fix is called the security update gap.
The way we're fixing vulnerabilities today is unfit for competing with attackers. The attackers have evolved tremendously in the last decade, improving on their ability to find vulnerabilities, analyze them, create exploits, and deliver those exploits faster and more efficiently than ever. They have specialized skills: Some write exploit kits, while others focus on attacks. They're running well-oiled businesses motivated by substantial profits.
The vulnerability-fixing side, in contrast, has been improving slowly and reluctantly, akin to a child sloppily cleaning his room just to tone down parents' constant complaining. Don't get me wrong here. With regular, semi-automatic updates provided by companies such as Microsoft, Adobe, and Oracle and update systems on Linux, Mac, Android, and iOS, we're in much better shape than we used to be when it comes to delivering fixes to users. But as long as we fail to make applying those fixes free of risk and hassle, attackers will continue to thrive inside the security update gap.
It's time to make pen testing harder
The fact that we seem to be getting worse at this over time has been so frustrating that we decided to do something about it—by making our jobs as pen testers harder. No, we're not insane. If we succeed, pen testing will become more expensive, but so will the cost of creating actual attacks.
We did this to help eliminate the four reasons why enterprise admins don't apply security fixes immediately once they get them. The four reasons are the following:
- Security updates are huge. They often replace hundreds of megabytes of code on computers and also include unnecessary functional fixes and undocumented changes that have a potential to break things. In addition, some industries require significant code changes to go through a formal approval process for compliance reasons.
- Security updates disturb users' productivity. They take five to ten minutes to install on each computer and usually require computers to be restarted.
- When something goes wrong with an update, it may be difficult to restore the original state. Imagine your update broke the laptop of a salesperson on another continent and you don't have a remote way of unbreaking it.
- Security updates may come in the form of a new version that does not work well with the rest of the system. This is a far too common issue. Just ask any enterprise why its computers are still full of ancient, vulnerable Java runtimes.
Here are my requirements for a better vulnerability-fixing system:
- Use tiny security patches, preferably just a few bytes, that are unlikely to have undesired side effects and can even be manually reviewed.
- Use hot patching. Instantly apply security patches to running processes so users aren't interrupted.
- Use hot unpatching. Instantly remove security patches from running processes when something goes wrong.
- Apply security patches to older versions of software when official updates don't exist for them.
Use micropatches to deploy fixes faster
Keeping these four requirements in mind, we went ahead and built a platform that lets us deliver tiny micropatches quickly and inexpensively to millions of computers. These can be hot-applied or hot-removed without disturbing users in any way. Think of it as digital microsurgery. We call this platform 0patch, and it just became publicly and freely available this month. 0patch makes it possible for security researchers to create micropatches for vulnerabilities they discover to either complement official vendor updates or provide a solution where there are no official fixes.
Imagine an enterprise admin who just received a new batch of security updates, including fixes for critical remotely exploitable vulnerabilities that attackers are already exploiting. His reasons for not immediately applying the huge official updates have not changed, so he will promptly start the process of their rigorous testing. But at the same time, he can apply a few micropatches to protect his computers until such time as testing is complete. To attackers' disappointment, he will effectively close the security update gap.
This sounds nice, but someone has to actually write these micropatches. We expect software vendors to gradually adopt micropatching, but we need to jump-start the concept. While our team has already created hundreds of micropatches, we are too small to make a serious dent. Even the largest security teams in the world are too small. If we want this to work, we need to engage the global community of security researchers. We need crowdpatching.
A call for crowdpatching
I'm therefore reaching out to all security researchers who want to not only find vulnerabilities, but also get them fixed quickly and painlessly. Crowdpatching includes many different activities, including:
- Hunting for proof-of-concept exploits. You can't patch what you can't demonstrate.
- Analyzing vulnerabilities. What goes wrong? Is it a buffer overflow, a use-after-free, a format string bug?
- Analyzing vendor updates. Extracting vulnerabilities from official updates will allow us to create "replacement micropatches" that administrators can apply while testing the official updates.
- Finding vulnerabilities in all kinds of products, especially the widely used libraries that are integrated in hundreds of products.
- Extracting vulnerabilities and exploits from exploit kits. Micropatching has the potential to put exploit kits out of business.
- Writing micropatches. Sometimes writing a micropatch is a trivial matter, other times it takes a deep understanding of the product's internals.
- Testing micropatches. These must be tested on real software, and getting access to it is sometimes difficult or expensive.
Many highly skilled researchers are finding vulnerabilities, dissecting them, and creating demo exploits. And many are frustrated that, having found a vulnerability, they will often be considered part of the problem by vendors and users instead of being recognized for the work they've done. Until now, these researchers had no good options as to what they could do with discovered vulnerabilities. Crowdpatching gives them a way to receive public recognition, create an immediately applicable patch, and present their work while it's still relevant.
Crowdpatching is not aimed against software vendors. Quite the opposite: While some vendors might initially dislike the idea of someone else fixing their code, they will undoubtedly agree that in pursuing the goal of protecting their users, we're on the same team. In fact, it is my personal goal to get vendors on board with micropatching so that, five to ten years from now, all new operating systems and major software will have native support for the micropatching of security bugs. Think of it as having a built-in digital immune system in every important piece of software.
When that happens, who do you think will write vendors' micropatches? The same people who join crowdpatching now. They will have the skills and the knowledge, and software vendors will either contract with them or hire them. We could even see specialized micropatching shops that write patches for vendors while the vendors continue to work on new products and features, rather than forcing their developers to drop everything each time a new vulnerability gets published.
Let the crowdpatching begin
The IT world has a growing problem with fixing software vulnerabilities, and our collective lack of action has made attackers' jobs easier each year. Unfortunately, all of the fancy security technology popping up on trade-show floors presents only a minor hurdle for offensive actors. The infosec community has the collective skills and knowledge required to fix vulnerabilities, but it lacks the technology to do so in an efficient enough way to actually make a difference. We're hoping to bringing such technology to the table for everyone to use.
Let the crowdpatching begin! Get involved by contacting email@example.com.