People attack, machines attack. The volume of attacks is huge, but the number of cyber security specialists has not grown as much as demand, a trend I looked at in my previous No Jitter article, "Combating the Security Specialist Drought." In the absence of qualified professionals, machine learning can be applied to assist specialists and maybe even discover attacks and their patterns that the specialists may miss.
I recently read a blog written by Mike Stute, Chief Scientist at Masergy, which discussed the idea that machine learning can be applied to cyber security. I was inspired to reach out to him to gain a bit more of his insight, and I presented him with a series of questions via email. Mike is a data scientist and the prime architect for the Unified Enterprise Security platform at Masergy. He does R&D in data science and code prototype analysis systems.
Three big issues come to mind:
Right now there is a serious shortage of qualified security analysts. This problem also extends to security programmers, who are often needed to integrate products. Due to demand, in both cases, a lot of people have taken certification classes to transition from networking or programming analysts into security analysts -- but this does not provide them with real-world experience. This lack of experience in dealing with issues and coordinating multiple departments to solve issues often leaves large gaps in actions and awareness.
Right now machine learning (ML) is making progress versus standard sandbox or virus scanning. To explain at a fairly technical level, a sandbox is created to run an unknown executable, and the changes it makes in the sandbox environment is used heuristically to determine if the executable is malware. Virus scanning is similar, but it uses files at rest. There are problems with both of these cyber security processes.
Many executables don't exist on their own, but instead wait for user input on when to stop execution (File->Close). Sandboxing must run applications for a given period of time, and then kill the application to examine the sandbox. This can be defeated by an application performing tasks that appear safe for a given period of time before executing the actual payload on the real host since the sandbox shuts down too early. The sandbox can also be detected by an executable examining the environment in a subtle way, and once it determines it is in a sandbox, act differently to appear benign. Since each version of the product has a given set of sandboxes, hackers write libraries of code that detect such sandboxes.
ML is now being used to replace heuristic analysis with a learned profile. Since ML can detect the features of malware, it no longer has to be a scoring system. Instead, security specialists can rely on the ability of ML to detect and provide a degree of certainty to the detection. Using this system, more positive detections can be found by features instead of exact matches or scored matches. This will help defeat polymorphic viral code at rest, instead of introducing the limitations of a sandbox. When most companies speak of ML, they are referring to this type of learning which solves a classification problem.
ML improves cyber security by providing detection based on hidden variables that are difficult for humans to analyze. In all cases, given enough time, a team of data scientists can find such relationships in data, but this is often a slow process. By using ML, we can allow a system to do this for us. That means different attacks can be detected based on previous data, even if the attacks use different vectors. In other words, it moves from "heuristics" that require human learning to teach the system how to grade behaviors, to using features in the data that are often deeper and harder to find. This can be performed much faster since no human analysis is required.
The second way we are improving security is by using ML to handle big data problems like network behavioral analysis. In reality, there are a very large number of variables to detecting network behaviors by looking at network traffic. It is an enormous data set and new correlations and data patterns are added every month. To data scientists, this means that the number of variables used for detection is always increasing, so algorithms must be automatically multivariate and even independent to the number of variables. This is best handled by ML techniques such as support vector machines and deep belief networks.
Most attacks are automated, as many hackers have libraries of attack tools they use. These tools have been developed by manually attacking systems that are local to the developer. I can attack my own system using a given version of the operating system or application, determine how this works, and create an automated tool to perform the action. This tool might be manually targeted, but once initiated it will automatically perform all actions. That is not to say that these tools are not used in sequence with the output of one tool determining the next one to use, but whenever possible even that can be automated.
However, the dangerous ones are the manual attacks. They are usually slow and easily lost in a sea of normal traffic. At least for now, human intelligence is far greater than the capabilities of any machine, though that is changing rapidly, and someone applying his or her experience and intelligence for the sole purpose of intrusion can be a serious event.
Next week, in part two of this interview, I'll talk to Mike about how ML can be applied to detect abnormal behavior, whether ML can adapt to automated attacks, and where Masergy fits into the equation.
