Service Behavior Profiling and Probabilistic Inference for Anomaly Detection

Network attacks that exploit network service vulnerabilities become popular in recent years. An attacker can remotely send malicious messages to a vulnerable service and gain the execution right to control the victim. Most of the Internet worms and part of the Botnet fall into this attack category, and such attacks often cause severe damages to our computers and network systems.
As we know, benign software would perform normal procedure to communication with a server to accomplish a network task collaboratively via predefined network protocols. Although malware takes similar actions to communicate with the server that it intends to compromise, malware behavior is not exactly the same as normal behavior. In our work, we design a novel anomaly detection framework targets on the attack vector of vulnerability exploitation on network service.
The key hypothesis to anomaly detection assumes anomalous behaviors are suspicious from a normality point of view. We focus on defining the notion of normality in a new perspective – network service – to detect anomalies. Once the definition of normality is specified, the violation of the normality (i.e., anomaly) is determined. We found certain abnormal communication procedures can be used to profile anomaly behavior. They are considered as the sign of an attack (i.e., attack symptom) when the attacker and the victim undergo sequences of compromising actions.
Past models often suffer from lacking of model normality verification, and they only focus on individual model. To confront the first issue, we show how to construct underlying protocol models by static and dynamic approach to guarantee the normality. For the latter issue, we combine multiple protocol/service models to construct a composite model for complex network services. We propose a method to construct composite service model with protocol interaction and correlation.
To build the normal protocol models for anomaly detection, we adopt the Principal Component Analysis (PCA) to analysis the normal behavior of a network protocol and extract the significant communication states. The PCA analyzes the real world network traffic traces and perform data classification to cluster different communication behaviors. Normal and significant behavior will be chosen to build the normal behavior model that is a form of finite state machine.
Our prototype system can statefully capture and monitor activities between hosts, and it progressively assesses possible network anomalies by multi-level behavior tracking, cross-level behavior triggering, and correlation of different network protocols and services.
To increase the confidence level of assessing attacks, we develop a probabilistic inference model to infer and compute the belief score of possible attacks based on the observation of the attack symptoms. In our observation, each attack symptom has a different degree of significance in the attack evaluation so that probability is an appropriate mathematical tool for attack inference.
We collect several real world attacks and build the normal protocol models that they use. Several anomalies and attack symptoms are detected by our system; no matter the attack is known, unknown, or a variant; even they do not exploit the same vulnerability.
The work has several novel research concepts. We focus on the network protocol and service as a basis to detect anomalies. We both adopt static and dynamic approach to build normal models. Using PCA to build normal model has not been seen in the past. Developing a cross-level monitoring system and composite service model are also new to this research field. The result shows our system can detect anomalies and is a good solution for intrusion detection.
Keyword: Anomaly detection, network service, behavior profiling, principal component analysis, inference model, finite state machine.